1. 12

    “You’re almost certainly affected by this extremely serious SQLite vulnerability we found, but we won’t tell you anything about it. Except by the way here’s the commit fixing it in Chromium. Except if you actually visit it all it does is upgrade to an entirely new SQLite version, so it doesn’t actually tell you anything useful. Also we pwd a very widely-used consumer product but we’re never going to tell you how, even after this is fixed.”

    I found this page very frustrating to read. They’re obviously giving vendors time to patch as if they were following the “usual” responsible disclosure procedures (they say they’re pushing vendors to patch in the first paragraph on the page), but if they’re just waiting for vendor coordination why even bother to announce at all? The page feels like it’s just there to build hype about this vulnerability because there’s nothing I can possibly do with it; I can’t examine the code, I can’t learn from the bug, I can’t patch my systems because distributions haven’t issued patches yet. What exactly is the point here? Is it literally just there to tell Google Chrome users to upgrade? (Because, at least on my Debian 9 systems, Chromium has not been fixed yet so only Chrome users can do anything.)

    1.  

      Yeah…

      Apparently the bug involves letting people run their own queries, which is probably not the case in most deployments (or you are already having a bad time), which can be crafted to corrupt the database.

      1. 12

        Untrusted code can run its own SQL queries against a SQLite database if it’s running in a browser which implements Web SQL. A couple do, still. ¯\_(ツ)_/¯

        Not all browsers ever implemented it at all and it’s deprecated now. The official reason given for deprecating it was that there wans’t a 2nd independent implementation besides SQLite. I suspect a supplementary reason might have been that someone thought giving “oh hey here’s a razor thin wrapper around raw SQLite, just go nuts with that complicated API provided by a huge pile of C code, I’m sure it will definitely not manifest any RCE issues” was unwise.

        1.  

          I suspect a supplementary reason might have been that someone thought giving “oh hey here’s a razor thin wrapper around raw SQLite, just go nuts with that complicated API provided by a huge pile of C code, I’m sure it will definitely not manifest any RCE issues” was unwise.

          that’s the core reasoning in this post

          1.  

            Aha! Thanks. I never saw that and I didn’t think it was ever spelled out explicitly in the standards docs.

        2. 6

          https://repo.or.cz/sqlite.git/blob_plain/HEAD:/test/fts3corrupt4.test

          There are some testcases that will crash unpatched sqlite versions (confirmed it using the “DB Browser for SQLite” Windows application). Turning that integer overflow into a heap corruption is an exercise to everyone except Tencent, basically.

          SQLite isn’t an API that’s designed well for running untrusted queries, yet here we are with Web SQL…

        3.  

          It’s interesting to me that people keep using SQLite as an exemplar arguing how C code can be safe. Even with its level of testing, there’s still RCE’s popping up. So, to answer your question, this announcement further corroborates the crowd saying to use safe, system languages. It also suggest these actions for SQLite itself:

          1. Don’t use it. Use a safer one instead. In realm of DB’s, there might not be a safer one. Search would start with an enumeration of SQL databases done in memory-safe languages. Preferably, concurrency-safe on top of that. If not, we at least have analyzers for some like Java.

          2. Use tech like Softbound+CETS that transforms existing C code into memory-safe code. That might block this one plus the next dozen.

          3. Use isolation architecture with trusted data separated from untrusted. Possibly two instances of the DB. Maybe add a monitor to the untrusted one to catch weird behavior.

          1.  

            From the post above yours:

            SQLite isn’t an API that’s designed well for running untrusted queries,”

            Except I’d leave out “well”. Sqlite is very solid for doing what it is designed to do. It is not designed as a public front end.

            1.  

              I agree with that. My point still stands on reliability, though, since the bugs found in testing and analysis were often preventable by a safe language like Ada.

              1. 5

                That list of “security vulnerabilities” is annoying as all hell. 99% of them are: someone built an application that takes untrusted user input and no permission control and incorporate into it a powerful SQL database and then decided to allow the untrusted users direct access to SQL commands and, shockingly, things are not safe. There are on the level of claiming the ADA compiler is unsafe because if you allow untrusted users to write and execute arbitrary ADA programs, someone bad can happen.

                1.  

                  Sqlite is very solid for doing what it is designed to do. It is not designed as a public front end.

                  Yeah, that’s what I meant. Executing untrusted queries just was never a design goal of SQLite. If that truly is your usecase, it’s probably better to use a database with a real user permission model so you can lock down the attack surface to something smaller than “the whole database.”

                  1.  

                    Great comparison haha.

          1.  

            Shocking to find out that giving raw sql access to untrusted users is bad security.

            1. 8

              yet in many respects, it is the most modern database management system there is

              It’s not though. No disrespect to PostgreSQL, but it just isn’t. In the world of free and open source databases it’s quite advanced, but commercial databases blow it out of the water.

              PostgreSQL shines by providing high quality implementations of relatively modest features, not highly advanced state of the art database tech. And it really does have loads of useful features, the author has only touched on a small fraction of them. Almost all those features exist in some other system. But not necessarily one single neatly integrated system.

              PostgreSQL isn’t great because it’s the most advanced database, it’s great because if you don’t need anything state of the art or extremely specialized, you can just use PostgreSQL for everything and it’ll do a solid job.

              1. 13

                but commercial databases blow it out of the water

                Can you provide some specific examples?

                1. 16

                  Oracle has RAC, which is a basic install step for any Oracle DBA. Most Postgres users can’t implement something similar, and those that can appreciate it’s a significant undertaking that will lock you into a specific workflow so get it right.

                  Oracle and MS-SQL also have clustered indexes. Not what Postgres has, but where updates are clustered as well. Getting Pg to perform sensibly in this situation is so painful, it’s worth spending a few grand to simply not worry about it.

                  Ever run Postgres on a machine with over 100 cores? It’s not much faster than 2 cores without a lot of planning and partitioning, and even then, it’s got nothing on Oracle and MS-SQL: Open checkbook and it’s faster might sound like a lose, but programmers and sysadmins cost money too! Having them research how to get your “free” database to perform like a proper database isn’t cost effective for a lot of people.

                  How about big tables. Try to update just one column, and Postgres still copies the whole row. Madness. This turns something that’s got to be a 100GB of IO into 10s of TBs of IO. Restructuring this into separate partitions would’ve been the smart thing to do if you’d remembered to do it a few months ago, but this is a surprise coming from commercial databases which haven’t had this problem for twenty years. Seriously! And don’t even try to VACUUM anything.

                  MS-SQL also has some really great tools. Visual Studio actually understands the database, and its role in development and release. You can point it at two tables and it can build ALTER statements for you and help script up migrations that you can package up. Your autocomplete can recognise what version you’re pointing at. And so on.

                  …and so on, and so on…

                  1. 3

                    Thanks for the detailed response. Not everyone has money to throw at a “real” enterprise DB solution, but (having never worked with Oracle and having only administered small MSSQL setups) I did wonder what some of the specific benefits that make a DBA’s life easier were.

                    Of course, lots of the open source tools used for web development and such these days seem to prefer Postgres (and sometimes MySQL), and developers like Postgres’ APIs. With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.

                    1. 7

                      Not everyone has money to throw at a “real” enterprise DB solution

                      I work for a commercial database company, so I expect I see a lot more company-databases than you and most other crustaceans: Most companies have a strong preference to rely on an expert who will give them a fixed cost (even if it’s “money”) to implement their database, instead of trying to hire and build a team to do it open-source. Because it’s cheaper. Usually a lot cheaper.

                      Part of the reason why: An expert can give them an SLA and has PI insurance, and the solution generally includes all costs. Building a engineering+sysadmin team is a big unknown for every company, and they usually need some kind of business analyst too (often a contractor anyway; more £££) to get the right schemas figured out.

                      Professional opinion: Business logic may actually be some of the least logical stuff in the world.

                      lots of the open source tools used for web development and such these days seem to prefer Postgres

                      This is true, and if you’re building an application, I’d say Postgres wins big. Optimising queries for dbmail’s postgres queries was hands down much easier than any other database (including commercial ones!).

                      But databases are used for a lot more than just applications, and companies who use databases don’t always (or even often) build all (or even much) of the software that interacts with the database. This should not be surprising.

                      With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.

                      I’m not sure I disagree, but I don’t think this is a good thing. EnterpriseDB isn’t Postgres. Neither is redshift. Queries that work fine in a local Pg installation run like shit in redshift, and queries that are built for EnterpriseDB won’t work at all if you ever try and leave. These kinds of “hybrid open source” offerings are an anathema, often sold below a sustainable price (and much less than what a proper expert would charge), leaving uncertainty in the SLA, and with none of the benefits of owning your own stack that doing it on plain postgres would give you. I just don’t see the point.

                      1. 3

                        Professional opinion: Business logic may actually be some of the least logical stuff in the world.

                        No kidding. Nice summary also.

                        1. 0

                          Queries that work fine in a local Pg installation run like shit in redshift

                          Not necessarily true, when building your redshift schema you optimize for certain queries (like your old pg queries).

                      2. 4

                        And yet the cost of putting your data into a proprietary database format is enough to make people find other solutions when limitations are reached.

                        Don’t forget great database conversion stories like WI Circuit Courts system or Yandex where the conversion to Postgres from proprietary databases saved millions of dollars and improved performance…

                        1. 2

                          Links to those stories?

                          1. 1

                            That Yandex can implement clickhouse doesn’t mean everyone else can (or should). How many $100k developers do they employ to save a few $10k database cores?

                            1. 2

                              ClickHouse has nothing to do with Postgres, it’s a custom column oriented database for analytics. Yandex Mail actually migrated to Postgres. Just Postgres.

                          2. 2

                            You’re right about RAC but over last couple of major releases Postgres has gotten alot better about using multiple cores and modifying big tables. Maybe not at the Oracle level yet bit its catching up quickly in my opinion.

                            1. 3

                              Not Oracle-related, but a friend of mine tried to replace a disk-based kdb+ with Postgres, and it was something like 1000x slower. This isn’t even a RAC situation, this is one kdb+ core, versus a 32-core server with Postgresql on it (no failover even!).

                              Postgres is getting better. It may even be closing the gap. But gosh, what a gap…

                              1. 1

                                Not to be that guy, but when tossing around claims of 1000x, please back that up with actual data/blogpost or something..

                                1. 6

                                  You remember Mark’s benchmarks.

                                  kdb doing 0.051sec what postgres was taking 152sec to complete.

                                  1000x is nothing.

                                  Nobody should be surprised by that. It just means you’re asking the computer to do the wrong thing.

                                  Btw, starting a sentence with “not to be that guy” means you’re that guy. There’s a completely normal way to express curiosity in what my friend was doing (he’s also on lobsters), or to start a conversation about why it was so much easier to get right in kdb+. Both could be interesting, but I don’t owe you anything, and you owe me an apology.

                                  1. 2

                                    Thanks for sharing the source, that helps in understanding.

                                    That’s a benchmark comparing a server grade setup vs essentially laptop grade hardware (quad-core i5), running the default configuration right out of the sample file from the Git repo, with a query that reads a single small column out of a very wide dataset without using an index. I don’t doubt these numbers, but they aren’t terribly exciting/relevant to compare.

                                    Also, there was no disrespect intended, not being a native english speaker I may have come off clumsy though.

                                    1. 1

                                      kdb doing 0.051sec what postgres was taking 152sec to complete.

                                      That benchmarks summary points to https://tech.marksblogg.com/billion-nyc-taxi-rides-postgresql.html which was testing first a pre-9.6 master and then a PG 9.5 with cstore_fdw. Seems to me that neither was fair and I’d like to do it myself, but I don’t have the resources.

                                      1. 1

                                        If you think a substantially different disk layout of Pg, and/or substantially different queries would be more appropriate, I think I’d find that interesting.

                                        I wouldn’t like to see a tuning exercise including a post-query exercise looking for the best indexes to install for these queries though: The real world rarely has an opportunity to do that outside of applications (i.e. Enterprise).

                                  2. 1

                                    Isn’t kdb+ really good at stuff that postgres (and other RDBMS) is bad at? So not that surprising.

                                    1. 1

                                      Sort of? Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C): Got some tall skinny table? Try mmaping as much as possible. That’s basically what kdb does.

                                      What was surprising was just how difficult it was to get that in Pg. I think we expected, with more cores and more disks it’d be fast enough? But this was pretty demoralising! I think the fantasy was that by switching the application to Postgres it’d be possible to get access to the Pg tooling (which is much bigger than kdb!), and we massively underestimated how expensive Pg is/can be.

                                      1. 3

                                        Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C)

                                        Well, kdb+ is columnar, which is pretty different than how most people approach naive database implementation. That makes it very good for some things, but really rough for others. Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).

                                        Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.

                                        But, yes, if your primary use cases favor a columnar approach, kdb+ will outperform vanilla postgres (as will monetdb, clickhouse, and wrappers around parquet files).

                                        You can get the best of both worlds You can get decent chunks of both worlds by using either the cstore_fdw or imcs extensions to postgres.

                                        1. 1

                                          which is pretty different than how most people approach naive database implementation.

                                          I blame foolish CS professors emphasising linked lists and binary trees.

                                          If you simply count cycles, it’s exactly how you should approach database implementation.

                                          Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).

                                          So I haven’t done that kind of UPDATE in any production work, but I also don’t need it: Every customer always wants an audit trail which means my database builds are INSERT+some materialised view, so that’s exactly what kdb+ does. If you can build the view fast enough, you don’t need UPDATE.

                                          Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.

                                          If I have data that I need horizontal grabs from, I arrange it that way in memory. I don’t make my life harder by putting it on the disk in the wrong shape, and if I do run into an application like that, I don’t think gosh using postgres would really speed this part up.

                              2. 3

                                Spanner provides globally consistent transactions even across multiple data centers.

                                Disclosure: I work for Google. I am speaking only for myself in this matter and my views do not represent the views of Google. I have tried my best to make this description factually accurate. It’s a short description because doing that is hard. The disclosure is long because disclaimers are easier to write than useful information is. ;)

                                1. 2

                                  @geocar covered most of what I wanted to say. I also have worked for a commercial database company, and same as @geocar I expect I have seen a lot more database use cases deployed at various companies.

                                  The opinions stated here are my own, not those of my former or current company.

                                  To put it bluntly, if you’re building a Rails app, PostgreSQL is a solid choice. But if you’ve just bought a petabyte of PCIe SSDs for your 2000 core rack of servers, you might want to buy a commercial database that’s a bit more heavy duty.

                                  I worked at MemSQL, and nearly every deployment I worked with would have murdered PostgreSQL on performance requirements alone. Compared to PostgreSQL, MemSQL has more advanced query planning, query execution, replication, data storage, and so on and so forth. It has state of the art features like Pipelines. It has crucial-at-scale features like Workload Profiling. MemSQL’s competitors obviously have their own distinguishing features and qualities that make them worth money. @geocar mentioned some.

                                  PostgreSQL works great at smaller scale. It has loads useful features for small scale application development. The original post talks about how Arcentry uses NOTIFY to great effect, facilitating their realtime collaboration functionality. This already tells us something about their scale: PostgreSQL uses a fairly heavyweight process-per-connection model, meaning they can’t have a huge number of concurrent connections participating in this notification layer. We can conclude Arcentry deployments using this strategy probably don’t have a massive number of concurrent users. Thus they probably don’t need a state of the art commercial database.

                                  There are great counterexamples where specific applications need to scale in a very particular way, and some clever engineers made a free database work for them. One of my favorites is Expensify running 4 million queries per second on SQLite. SQLite can only perform nested loop joins using 1 index per table, making it a non-starter for applications that require any kind of sophisticated queries. But if you think about Expensify, its workload is mostly point look ups and simple joins on single indexes. Perfect for SQLite!

                                  1. 1

                                    But MemSQL is a distributed in-memory database? Aren’t you comparing apples and oranges?

                                    I also highly recommend reading the post about Expensify usage of SQLite: it’s a great example of thinking out of the box.

                                    1. 1

                                      No. The author’s claims “Postgres might just be the most advanced database yet.” MemSQL is a database. If you think they’re apples and oranges different, might that be because MemSQL is substantially more advanced? And I used MemSQL as one example of a commercial database. For a more apples-to-apples comparison, I also think MSSQL more advanced than PostgreSQL, which geocar covered.

                                      And MemSQL’s in-memory rowstore serves the same purpose as PostgreSQL’s native storage format. It stores rows. It’s persistent. It’s transactional. It’s indexed. It does all the same things PostgreSQL does.

                                      And MemSQL isn’t only in-memory, it also has an advanced on-disk column store.

                              1. 1

                                As Wolfram pointed out, what academic computer scientists mean by “type theory” doesn’t correspond at all with mathematical use which is much more flexible and rich. Academic computer science confuses metamathematical ideas with mathematical practice.

                                1. 2

                                  But unruly and violent or not, the movement is popular: 72% of the country currently supports it. French unions are announcing their support. And they’ve now also forced President Macron into a corner. France’s prime minister, Édouard Philippe, announced Tuesday that fuel tax increases would be suspended for six months as the country deals with the situation.

                                  If 72% of the population supports violent protests, the anger was already there. Maybe that specific change to facebook’s algorithm happened to spark it, but so could a hundred other things.

                                  1. 1

                                    The interesting thing is the huge platform for just nutty angry stuff: anti-vaxxers, fake stories,…

                                    According to Ludosky’s Facebook page, she spent the summer on Facebook sharing links to various aromatherapy products and promoting her Change.org petition.

                                    or

                                    According to the about page for “Citizens in Anger,” its true goal is to defend France against a Masonic cabal of global bankers who control France. It currently has 15,000 members.

                                  1. 2

                                    My understanding is that Fortran is used pretty heavily in heavily-paralisable workflows instead of C (mainly because of the array ops I think?). In a research lab I stayed in for a bit, there were people in the high performance computing corner doing fortran stuff…sometimes. I don’t know the full extent of this effect though

                                    1. 1

                                      That is the theory - I would love to see some studies of how well this actually works. Dennis Ritchie was dubious.

                                      1. 2

                                        What a nice pick of a video, even though the context really only unfolds at the very end.

                                      1. 20

                                        My sense now is that Alan Kay’s insight, that we can use the lessons of biology (objects are like cells that pass messages to each other), was on target but it was just applied incompletely. To fully embrace his model you need asynchrony. You can get that by seeing the processes of Erlang as objects or by doing what we now call micro-services. That’s where OO ideas best apply. The insides can be functional.

                                        1. 17

                                          “If you want to deeply understand OOP, you need to spend significant time with SmallTalk” is something I’ve heard over and over throughout my career.

                                          1. 5

                                            It’s also a relatively simple language with educational variants like Squeak to help learners.

                                            1. 7

                                              I have literally taken to carrying around a Squeak environment on USB to show to people. even experienced engineers tend to get lost in it for a few hours and come out the other side looking at software in a different way, given a quick schpiel about message passing.

                                            2. 4

                                              If you don’t have any Smalltalk handy, Erlang will do in a pinch.

                                              1. 2

                                                And if you don’t have Erlang handy, you can try Amber in your browser!

                                              2. 1

                                                I went through the Amber intro that /u/apg shared. I’d love to dive deeper. If anyone has any resources for exploring SmallTalk/Squeak/Etc further, I’d love to see them. Especially resources that explore what sets the OO system apart.

                                                1. 2

                                                  I’m told that this is “required” reading. It’s pretty short, and good.

                                              3. 16

                                                I even wrote a book on that statement. My impression is that “the insides can be functional” could even be “the insides should be functional”; many objects should end up converting incoming messages into outgoing messages. Very few objects need to be edge nodes that turn incoming messages into storage.

                                                But most OOP code that I’ve seen has been designed as procedural code where the modules are called “class”. Storage and behaviour are intertwingled, complexity is not reduced, and people say “don’t do OOP because it intertwingles behaviour and storage”. It doesn’t.

                                                1. 2

                                                  This.

                                                  Whether the implementation is “functional” or not, the internals of any opaque object boundary should at least be modellable as collection of [newState, worldActions] = f(old state, message) behaviours.

                                                  We also need a unified and clearer method for namespacing and module separation, so that people aren’t forced to make classes (or closures-via-invocation) simply to split the universe into public and private realms.

                                                  To say that the concept of objects should be abandoned simply because existing successful languages have forced users to mis-apply classes for namespacing is as silly as the idea that we should throw out lexical closures because people have been misusing them to implement objects (I’m looking at you, React team).

                                                2. 5

                                                  If there’s one lesson I’ve learned from software verification, it’s that concurrency is bad and we should avoid it as much as possible.

                                                  1. 8

                                                    I’m not entirely sure this is correct. I’ve been using Haskell/Idris/Rust/TLA+ for a while now and I’m now of the opinion that concurrency is just being tackled at the wrong conceptual level. In that most OOP/imperative strategies mix state+action when they shouldn’t.

                                                    Also can you qualify what you mean by concurrency? I’m not sure if you’re conflating concurrency with parallelism here.

                                                    I’m using the definitions offered by Simon Marlow of Haskell fame, from Parallel and Concurrent Programming in Haskell:

                                                    In many fields, the words parallel and concurrent are synonyms; not so in programming, where they are used to describe fundamentally different concepts.

                                                    A parallel program is one that uses a multiplicity of computational hardware (e.g., several processor cores) to perform a computation more quickly. The aim is to arrive at the answer earlier, by delegating different parts of the computation to different processors that execute at the same time.

                                                    By contrast, concurrency is a program-structuring technique in which there are multiple threads of control. Conceptually, the threads of control execute “at the same time”; that is, the user sees their effects interleaved. Whether they actually execute at the same time or not is an implementation detail; a concurrent program can execute on a single processor through interleaved execution or on multiple physical processors.

                                                    While parallel programming is concerned only with efficiency, concurrent programming is concerned with structuring a program that needs to interact with multiple independent external agents (for example, the user, a database server, and some external clients). Concurrency allows such programs to be modular; the thread that interacts with the user is distinct from the thread that talks to the database. In the absence of concurrency, such programs have to be written with event loops and callbacks, which are typically more cumbersome and lack the modularity that threads offer.

                                                    1. 5

                                                      Also can you qualify what you mean by concurrency?

                                                      Concurrency is the property that your system cannot be described by a single global clock, as there exist multiple independent agents such that the behavior the system depends on their order of execution. Concurrency is bad because it means you have multiple possible behaviors for any starting state, which complicates analysis.

                                                      Using haskell/rust/Eiffel here helps but doesn’t eliminate the core problem, as your system may be larger than an individual program.

                                                      1. 10

                                                        All programs run in systems bigger than the program

                                                        1. 1

                                                          But that’s not an issue if the interaction between the program and the system is effectively consecutive (not concurrent), I think is the point that was being made. A multi-threaded program, even if you can guarantee is free of data races etc, may still have multiple possible behaviors, with no guarantee that all are correct within the context of the system in which operates. Analysis is more complex because of the concurrency. A non-internally-concurrent program can on the other be tested against a certain input sequence and have a deterministic output, so that we can know it is always correct for that input sequence. Reducing the overall level of concurrency in the system eases analysis.

                                                          1. 2

                                                            You can, and probably should, think of OS scheduling decisions as a form of input. I agree that concurrency can make the state space larger, but I don’t believe it is correct to treat concurrency/parallelism as mysterious or qualitative.

                                                        2. 3

                                                          Using haskell/rust/Eiffel here helps but doesn’t eliminate the core problem, as your system may be larger than an individual program.

                                                          They help in reducing the scope into the i/o layer interacting with each other. I think an example would be helpful here as there isn’t anything to argue for your stated position so far.

                                                          But lets ignore language for the moment and give an example from my work. We have a network filesystem that has to behave generally like a POSIX filesystem across systems. This is all c and in kernel, so mutexes and semaphores are the overall abstractions in use for good or ill.

                                                          I’ve been using TLA+ both as a learning aide in validating my understanding of the existing code, and to try to find logic bugs in general for things like flock() needing to behave across systems.

                                                          Generally what I find is that these primitives are insufficient for handling the interactions in i/o across system boundaries. Aka lets take a call to flock() or even fsync(), you need to ensure all client systems behave in a certain way when one (or more) systems make a call. What I find is that the behavior as programmed works in general cases, but when you setup TLA+ to mimic the mutex/semaphores in use and their calling behavior, they are riddled with logic holes.

                                                          This is where I’m trying to argue that the abstraction layers in use are insufficient. If we were to presume we used rust in this case, primarily as its about the only one that could fit a kernel module use case, there are a number of in node concurrent races across kernel worker threads that can just “go away”. Thus freeing us to validate our internode concurrent behavior logic via TLA+ and then ensuring our written code conforms to that specification.

                                                          As such, I do not agree that concurrent programming should be avoided whenever possible. I only argue that OOP encourages by default bad practices that one would want to use when programming in a concurrent style (mixing state+code in an abstraction that is ill suited for it). It doesn’t mean OOP is inherently bad, just a poor fit for the domain.

                                                          1. 1

                                                            I feel that each public/private boundary should have its own singular clock, and use this to sequence interactions within its encapsulated parts, but there can never really be a single global clock to a useful system, and most of our problems come from taking the illusion of said clock further than we should have.

                                                        3. 4

                                                          I would go exactly tangential and say that the best software treats concurrency as the basis of all computation. in particular, agnostic concurrency. if objects are modeled to have the right scope of visibility and influence, they should be able to handle messages in a perfectly concurrent and idempotent manner, regardless of cardinality.

                                                          1. 2

                                                            Take Clojure for example, and I think concurrency is not that bad, and there is no reason to avoid it. Mutability and intertwining of abstractions is what leads to problematic situations. Functional programming solves that by its nature.

                                                            1. 4

                                                              Even if the program is immutable, you ultimately want the program to have some effect on the outside world, and functional programming doesn’t magically fix the race conditions there. Consider having a bunch of immutable, unentwined workers all making HTTP requests the same server. Even if there are no data races, you can still exceed the rate limit due to concurrency.

                                                        1. 33

                                                          I’ve run into this mentality myself a couple of times, people claim in 2018 that they can write safe C/C++, it’s only those other people that can’t.

                                                          1. 7

                                                            I would claim that 2018 is the best time (yet) for writing memory-safe C++. We have tooling and paradigms now where we don’t need to rely on raw pointers and can make use of things like shared_ptr and unique_ptr. We can utilize the vector class which gives us OOB write protection (via the vector::at method) . We can even look to the design decisions made in Rust and see they draw their roots from modern C++ (think RIAA). All the protections of Rust already exist in C++, and are distinct enough that tooling can catch when developers don’t use them.

                                                            1. 20

                                                              I agree about it being best time to write safer C++. Ill add C, too, given all the analyzers. Empirical evidence Gaynor references show vast majority of C/C++ code fails to be safe anyway. So a safe-by-default option is better to use with unsafety turned on selectively when necessary.

                                                              Also, the key aspects of Rust’s memory safety come from Cyclone language. It was a safer C with some temporal safety among other things. Clay was another which had linear and singleton types.

                                                              1. 15

                                                                All the protections of Rust already exist in C++

                                                                Unless you are claiming C++ has a way to ensure data race freedom, this does not seem true.

                                                                1. 10

                                                                  Smart pointers have been around for more than a decade (in Boost before they got into std::), and STL has been around for ages of course. From the memory safety perspective, the tools have been around for a long time. Perhaps the issue is that none of these things are mandatory.

                                                                  1. 4

                                                                    Halfway there: C++ was built on unsafe-by-default foundation, C, with good protections added that also aren’t mandatory. It evolved from there in a lot of ways I didnt keep up with. I just know that plus its parsing/semantic headaches gave C++ a brutal start if goal was safety on average or high assurance.

                                                                  2. 9

                                                                    (via the vector::at method)

                                                                    Which you have to remember to use instead of the far easier nicer looking []. I’ve seen v.at in a C++ codebase exactly once in my life. As usual, the default C++ is unsafe.

                                                                    We can even look to the design decisions made in Rust and see they draw their roots from modern C++ (think RIAA)

                                                                    True but not sure of the relevance.

                                                                    All the protections of Rust already exist in C++

                                                                    They most certainly do not.

                                                                    tooling can catch when developers don’t use them

                                                                    There are no tools that I’m aware of that guarantee an absence of bugs in a C++ codebase. I’ve used most of them. Bugs still happen. Bugs that don’t happen in safer languages.

                                                                    1. 8

                                                                      All the protections of Rust already exist in C++

                                                                      I’m not sure that’s entirely true, but I could be wrong. I agree with what you are saying about RIAA, but a few other things pop to mind:

                                                                      • Rust tries really hard to ensure you have either only immutable references, or a sole mutable reference. This prevents the old “iterate over a collection and mutate it” problem.

                                                                      • It also forces you to think about sharing semantics across threads (i.e. Sync trait).

                                                                      • In Rust you can design APIs which “consume” their parameters. i.e. you pass ownership to a callee, and it never hands it back. This is useful for scenarios where you don’t want the user to try and re-use a conceptually finalized data structure. Perhaps reuse of the structure could be unsafe for example.

                                                                      I’m sure there will be other examples. And I’m sure the proponents of Rust will provide those in more comments ;)

                                                                      Maybe you can do these kinds of things in modern C++?

                                                                      1. 6

                                                                        Problem with C & C++ is not only about memory safety. Are you 100% sure you have absolutely no chance of a signed int overflow anywhere in your codebase?

                                                                        1. 15

                                                                          Compile with -fwrapv to instantly unlock the same behavior as rust.

                                                                          1. 5

                                                                            To get the K&R C behavior not the house-of-horrors C standard.

                                                                            1. 3

                                                                              Or better, -ftrapv to trap even errors that Rust won’t catch. (I don’t understand why this wasn’t made the default behaviour in Rust, to be honest; the performance hit should ultimately be negligible, and they already make some choices for safety-over-performance).

                                                                              1. 4

                                                                                The reason it is not default in Rust is that performance hit was not negligible.

                                                                                1. 0

                                                                                  Got some numbers? Part of the problem is probably that the code generator (LLVM) doesn’t handle it very well; but basing a language decision on a current limitation of the code generator doesn’t seem wise. (Especially if it’s a choice between “fail to detect some potentially critical code issue” and somewhat reduced performance).

                                                                                  1. 5

                                                                                    I don’t have the reference for Rust handy, but NIST Technical Note 1860 is one good reference. To quote the abstract:

                                                                                    We performed an experiment to measure the application-level performance impact of seven defensive code options… Of the seven options, only one yielded clear evidence of a significant reduction in performance; the main effects of the other six were either materially or statistically insignificant.

                                                                                    That one option is -ftrapv.

                                                                                    1. 1

                                                                                      That report is specific to GCC, though, which has worse overflow checking even that LLVM.

                                                                                      “In practice, this means that the GCC compiler generates calls to existing library functions rather than generating assembler instructions to perform these arithmetic operations on signed integers.”

                                                                                      Clang/LLVM doesn’t do that. Needless to say, it explains much of the performance hit, but not how it would affect Rust. I’m curious about what the real numbers do look like.

                                                                                    2. 1

                                                                                      (actually, having done some recent ad-hoc tests LLVM seems to do ok, mostly just inserting a jo instruction after operations that can overflow, which is probably about as good as you can do on x86-64. I’d still like to see some numbers of how this affects performance, though).

                                                                              2. 2

                                                                                I agree memory safety is not the only kind of security bug, but it is the focus of this article, which is why I focused on it in my comment.

                                                                            2. 4

                                                                              And how do you know they’re wrong?

                                                                              We see evidence all the time. Easily preventable bugs in old software everybody used but nobody wanted to make good. Bugs that would’ve been obvious if people had looked at compiler warnings or run a static analyzer, but nobody did..

                                                                              1. 24

                                                                                And how do you know they’re wrong?

                                                                                I don’t know about @szbalint, but I know they’re wrong because of ample experience with people who think they can write safe C code and can’t. I literally have never met or have worked with an engineer I’d trust to write C code, myself obviously included.

                                                                                I’ve been compiling all C and C++ code with -Wall for at least the last 15 years, and the last 10 with -Wall -Wextra -Werror. I’ve used free and paid-for static analysers on production code. Tests, extensive code reviews, valgrind, clang sanitizers, the lot.

                                                                                The end result were bugs, bugs, and more bugs, many of them security vulnerabilities. This problem isn’t solvable with developer discipline and tools. We’ve tried. We’ve failed.

                                                                                1. 2

                                                                                  Has your experience with D been much better? I’ve been able to segfault D because I didn’t realise that SomeClass c; generates a null pointer. I really wish the language were modified to make this impossible without some extra attributes like @trusted.

                                                                                  That’s the only way I’ve managed to really “crash” D, though. The rest of the time I get pretty stack traces at runtime whenever I mess up, much more controlled and predictable error-handling.

                                                                                  1. 1

                                                                                    Has your experience with D been much better?

                                                                                    Vastly so.

                                                                                    I’ve been able to segfault D because I didn’t realise that SomeClass c; generates a null pointer.

                                                                                    That’s not considered an issue in D. Variables are all default-initialised so you’ll get a crash at the point where you forgot to assign the reference to a new instance. Deterministic crashes in the face of bugs is a good thing.

                                                                                    1. 1

                                                                                      A segfault is a pretty obscure thing, and I’m not sure it’s even exactly “deterministic”. Are all arches guaranteed to react the same way to dereferencing a null pointer? It’s not like it’s raising a D NullPointerException, if such a thing even exists, so I don’t get a stack trace telling me where the problem is, unlike every other D “crash”.

                                                                                      I still don’t understand why making this segfaulting easy is desirable. It feels like an obvious rake to step on.

                                                                                      1. 1

                                                                                        A segfault is a pretty obscure thing

                                                                                        To a scripter, maybe, but not to a systems programmer.

                                                                                        and I’m not sure it’s even exactly “deterministic”

                                                                                        Some are, some aren’t. The deterministic ones are of the good variety. Always crashing because of an unitialised reference is ok. Getting a segfault somewhere else in your app isn’t.

                                                                                        It’s not like it’s raising a D NullPointerException, if such a thing even exists

                                                                                        It doesn’t exist, or rather, its version is to just segfault.

                                                                                        so I don’t get a stack trace telling me where the problem is, unlike every other D “crash”.

                                                                                        % coredumpctl gdb
                                                                                        % bt
                                                                                        

                                                                                        If you’re not running systemd then run the program again under gdb. Since it always crashes in the same place you’ll get the same result. If you’re on Windows you open the debugger right there and then and look at the backtrace.

                                                                                        It feels like an obvious rake to step on.

                                                                                        It isn’t, it always takes me 30s to fix.

                                                                                  2. 2

                                                                                    I don’t know what your use involves, but sometimes using e.g. Coverity is not much better than never using it. All those things have to be done in the context of a comprehensive build and test system and always done on every checkin. And there is no substitute for good programmers with solid education. I have met a number of really excellent C programmer who produce solid, trustworthy code.

                                                                                    And, as always in such discussions, none of these assertions make sense without a “compared to”. People write unsafe C code, compared to which people who write what?

                                                                                    1. 2

                                                                                      using e.g. Coverity is not much better than never using it

                                                                                      Not in my experience.

                                                                                      And there is no substitute for good programmers with solid education.

                                                                                      Few and far between.

                                                                                      I have met a number of really excellent C programmer who produce solid, trustworthy code.

                                                                                      I have not. And that includes meeting compiler writers and C++ luminaries.

                                                                                      People write unsafe C code, compared to which people who write what?

                                                                                      To people who write code in Ada, Rust, …

                                                                                    2. 1

                                                                                      The end result were bugs, bugs, and more bugs, many of them security vulnerabilities. This problem isn’t solvable with developer discipline and tools. We’ve tried. We’ve failed.

                                                                                      Then why is it that when the bugs that relate to language level issues are dissected in the public, they nearly always seem like they’d be prevented by discipline & disciplined application of tooling?

                                                                                      1. 4

                                                                                        Even the best programmers have an error rate. Much lower than novice programmers, but it’s still there.

                                                                                        1. 1

                                                                                          You are stating the obvious here.

                                                                                          The question is, why aren’t these errors being caught when it is so easy to do?

                                                                                          1. 6

                                                                                            The nature of the tooling seems like a prime candidate answer to that question. In C and C++ you often need to opt into constructs and static analyses in order to guard against memory safety bugs. Just look at all the tortured defenses in this thread alone. It’s some combination of “oh just use this particular subset of C++” all the way to “get gud.”

                                                                                            In memory safe languages the tooling generally demands that you opt out in order to write code that is susceptible to memory safety bugs.

                                                                                            It’s just another instance of “defaults matter.”

                                                                                            1. 0

                                                                                              I think we’re going in circles here. None of what you say is evidence that people can’t write safe code. You’re just saying they (most of them) won’t, which isn’t the contentious point.

                                                                                              1. 5

                                                                                                Seems like a distinction without a practical difference. You asked a question, “why aren’t these errors being caught when it is so easy to do,” and I answered with, essentially, “maybe it’s not as easy as you think.” I proposed a reason why it might not be easy.

                                                                                            2. 4

                                                                                              The question is, why aren’t these errors being caught when it is so easy to do?

                                                                                              It isn’t easy. If it were, they’d be caught. The litany of security vulnerabilities everywhere shows us that we’re doing it wrong.

                                                                                              1. 1

                                                                                                Right. Why not?

                                                                                            3. 2

                                                                                              Did I claim developer discipline wouldn’t prevent bugs? What I’m claiming is that it’s humanly impossible to manage that level of discipline in any large codebase. And I’m claiming that because we have decades of experience telling us that it’s the case.

                                                                                          2. 3

                                                                                            You raise a good point. I think anyone who thinks their own code is perfect is wrong, but I can’t prove it. I suppose, also, that it could be vacuously true in the sense that maybe their own code is perfect, but they haven’t written production systems entirely by themselves. Was that what you were suggesting?

                                                                                            1. 2

                                                                                              You raise a good point. I think anyone who thinks their own code is perfect is wrong, but I can’t prove it.

                                                                                              Can you write one line of code that is perfect? Why not two? Why not ten? Why not a thousand?

                                                                                              Even then, there’s a continuum between perfect and unmitigated disaster. I’ll grant you that I don’t really believe in large scale software ever being perfect in a literal sense of the word. Not even bug free. However, I believe there are individuals who can produce safe software in C. And if given enough control over the platform the code runs on, they can lock things down to virtually guarantee that language level issues are not ever going to turn into full compromise (RCE) or secret leaks.

                                                                                              If you need something for a case study, why not take a look at qmail? The fun thing is that the papers and writeups about qmail’s security practices don’t really mention things such as extensive use of static analyzers, fuzzers, and formal verification. Despite that, it has an incredible track record. I think there is much more that could be done to raise the bar.

                                                                                              I suppose, also, that it could be vacuously true in the sense that maybe their own code is perfect, but they haven’t written production systems entirely by themselves. Was that what you were suggesting?

                                                                                              Security costs time and isn’t sexy. Worst of all, you can’t measure it like you can measure performance or the length of the list of features. So even if someone out there is producing “perfect” code, it’s likely that the project goes mostly unheard of. If one were to dig up and point out such a project, people would just call it a toy / academic exercise / “nobody uses it.” People might say they care about security but they really don’t, they just use whatever is convenient or popular. And when you point out the bugs in that, well, developers being developers, they just pat each other on the back. “Dude, bugs happen! Give ’em a break!”

                                                                                              It’s especially bad in any commercial setting, which is why I think it is indeed the case that the people who are capable of writing secure software are, in the end, not doing that, because they don’t get to write an entire production system on their own terms. I don’t think it’s a coincidence that the project I just mentioned is essentially a solo project by one person.

                                                                                              I’m in that boat too, sort of. At day job there’s so much code that’d get you kicked out if it were my personal project with a security guarantee. I’m not at liberty to rearchitect and rewrite the software. The markets opt out of security.

                                                                                              1. 6

                                                                                                Can you write one line of code that is perfect? Why not two? Why not ten? Why not a thousand?

                                                                                                It depends on the line. Perfection isn’t just about the semantics the code has to the compiler, but about how a future reader will interpret it. Even the shortest snippet of code can have subtleties that may not be obvious later, no matter how clear they seem to me today. Concretely, in the 90s, “everyone knew” that system() was a huge security vulnerability because of how it adds to the attack surface, and it wasn’t a big deal because code that was thought of as needing to be secure was hardened against it. But those very same semantics came as a very unwelcome surprise to “everyone” when Shellshock was publicized in 2014.

                                                                                                Lots of vulnerabilities can be traced to people misunderstanding single lines of code.

                                                                                                I very much agree with your point about it being hard to sell security. I think that’s by far the biggest factor in how horrible the current state of affairs is.

                                                                                                1. 3

                                                                                                  Wasn’t the key shellshock problem that bash executed environment variables? 99% of these failures seem to come from parsing errors, eval, and convenience components. Why bash designers felt it was good to allow arbitrary functions to be passed in environment variables and then executed baffles me but probably came from feature creep. The same functionality could have been achieved more safely by include/load like mechanisms (not 100% safe either, but easier to lock down) or, better, running other programs.

                                                                                                  BTW: Bash scripts are memory safe.

                                                                                                  1. 2

                                                                                                    It depends on what you see as the most unexpected part of it. The reason it was such an emergency to patch it was that bash was exposed to unauthenticated incoming network connections in the first place.

                                                                                          3. -3

                                                                                            Where are all the safe Rust based applications and operating systems?

                                                                                              1. 6

                                                                                                I mean, so what? The pool of talented systems programmers is vanishingly small, and also going to be dominated by people using C/C++, so its not that surprising that it hasn’t eradicated all the competition yet. And as a sibling commenter pointed out, its doing pretty well for being all of 3 years old.

                                                                                                1. 0

                                                                                                  There is a big difference between “has not eradicated the competition yet” and “still can’t point to 5 widely used and superior applications”.

                                                                                                  1. 10

                                                                                                    how many did python have within 3 years? Or ruby for that matter. How about clojure? None of those had “5 widely used and superior applications” within 3 1/2 years. It was probably a decade before python had 5, maybe a little less for ruby. And you didn’t respond at all to my point about the quantity of systems programmers. Rust is a kind of weird language, attempting to fill a very tight ecological niche with few potential adoptees in terms of actual programmers. I’m not in the least surprised that its still very much in an embryonic state in regards to a community and the works that would flow from that. And, fwiw, Servo and Fuscia aren’t exactly small scale applications, not to mention all of the companies using it internally for whom we have no data/reports other than job board postings.

                                                                                                    1. 1

                                                                                                      I’m not at all saying Rust is a failure - I don’t find it appealing, but that doesn’t mean anything. But it’s still in the experiment stage. You can declare C obsolete when you have that body of large successful, less buggy, products to point at.

                                                                                                      1. 9

                                                                                                        The whole point of the article is that literally decades of experience have shown that it is effectively impossible to write safe secure code in C/C++. The goal in fact is to to declare C/C++ a security nightmare. Does that make it obsolete? Maybe, maybe not. There are options out there that give you far more safety guarantees than C/C++ will ever be able to do.

                                                                                                        Knowing that it is quite possibly flat our irresponsible to use it for any new project where security is important. Oh, and security is nearly always an important concern for a software project.

                                                                                                        1. 1

                                                                                                          Let me try again: To make that claim be anything more than marketing, you’d need some (a) data and (b) some indication that C programs were WORSE than some alternative. In the absence of both, it’s just weak marketing. The evidence seems to show more that it is very difficult to develop large software systems that are safe and secure and, at the same time, find an appreciative audience - no matter what the language. As I pointed out before, until there are significant examples of safe/secure/successful Rust programs to compare with, such claims are just blowing smoke.

                                                                                                          1. 2

                                                                                                            While it’s not as strong a claim as you seem to want, I don’t think it’s incorrect to say that C and C++ in the hands of not superhuman developers tends to result in a whole class of bugs that Rust makes nearly impossible to create. Bugs that have appeared numerous times and continue to keep appearing in critical internet infrastructure.

                                                                                                            It’s purely anecdotal but Rust has prevented multiple use after free, use before initialized, and buffer overrun errors in my own code multiple times just while playing around. It’s a bit disingenuous to suggest that C/C++ don’t have a problem and that Rust which provably prevents most of those problems isn’t a promising solution.

                                                                                                            This isn’t blowing smoke it’s a recognition that there is a problem and possible solution to that problem that doesn’t involve cloning Dan Bernstein and firing all the other programmers.

                                                                                                            1. 2

                                                                                                              I kind of like the idea of cloning Dan Bernstein and firing all the other programmers. It’s a big idea with some verve and panache.

                                                                                                            1. 4

                                                                                                              I agree the comment overstated by saying “effectively impossible.” Really hard with higher failure rates than safe languages would be more accurate. Yet, it seems misleading for you to use Bernstein and Hipp since most people are nowhere near as good at QA as them. Bernstein is also a security genius.

                                                                                                              C defenders countering with the vulnerability results of security geniuses in minimalist apps instead of what average C coder achieves vs average user of safe language isnt a good comparison.

                                                                                                              1. 0

                                                                                                                Hence my quest for examples rather than handwaving. Examples of real applications that are not minimalist. Specific examples.

                                                                                                                1. 2

                                                                                                                  Ripgrep? Firefox? The former is shipped and used for search in VS Code, the latter is pretty rapidly moving to using rust where it can to improve performance, and reduce security issues. Just two projects I can name off the top of my head as a non rust programmer.

                                                                                                                  1. 0

                                                                                                                    The same two projects everyone names and I’m not dismissing either of them. I’m just pointing out that the triumphal declarations of the obsolescense of C and dawn of the Reign of Rust lack sufficient backing. If we just got enthusiastic reports about what people wrote in Rust and how well it worked that would be interesting and impressive. But this overblown marketing stuff is just irritating.

                                                                                                      2. 5

                                                                                                        Well, I guess ripgrep qualifies? Hopefully there will be more such applications.

                                                                                                        1. -1

                                                                                                          Ripgrep is interesting. Is it considerably less bug ridden than ag ? Is it considerably less bug ridden than grep?

                                                                                                          https://www.cvedetails.com/product/23804/GNU-Grep.html?vendor_id=72 Well grep doesn’t seem so bad.

                                                                                                          Come on, people need to do better than this.

                                                                                                          1. 7

                                                                                                            Is it considerably less bug ridden than ag ?

                                                                                                            Yes, by a very very large margin across at least a couple different spectrums.

                                                                                                            Is it considerably less bug ridden than grep?

                                                                                                            Unlikely.

                                                                                                            Come on, people need to do better than this.

                                                                                                            Right. Nothing will ever be good enough. This is a classic Internet debate tactic. No matter what example anyone gives you, there will always be a reason to dismiss it. I don’t say that flippantly necessarily. Satisfying your standard (mentioned in another comment) to a high degree is nearly impossible. There will always be differences and variables that cannot reasonably be controlled for.

                                                                                                            It’s totally fine to have high standards that are impossible to satisfy (it’s your choice), but at least state that up front explicitly and don’t be coy about it. And don’t accuse people who are trying to hit a lower standard of evidence as just “blowing smoke,” because that’s a bunch of bullshit and you know it.

                                                                                                            1. -2

                                                                                                              Oh come on: ripgrep is the example I’m given. Yet, grep, written in horrible pathetic C, has very few security bugs. Are there CVEs for ag even? I am ready to be persuaded and I don’t need ironclad proofs - I just want to see a number of examples of complex applications/systems written in XYZ that are significantly better than the standard C versions in terms of security. Until those are available it’s just marketing. Maybe Rust is a brilliant advance that will lead to the creation of highly reliable secure applications - maybe it is not. Without evidence, all this claim that C has been obsoleted by Rust or whatever is exactly that - just blowing smoke. Read software jewels by Parnas, this is not a new problem.

                                                                                                              I’ll give you a good example: the OP makes a big deal about security issues in ImageMagick! Of all things. I cannot imagine that anyone involved in the development of ImageMagick worried about security at all - it was a tool people could use to manipulate images. Now it is being dragged into service as an online utility exposed to the open internet or used on images that come from anywhere and - lo and behold - because of C it’s insecure! Great, let me see a Rust program designed without any attention to security employed similarly and we’ll compare.

                                                                                                    2. 1

                                                                                                      Quite a few swift based applications out there though, so it isn’t like new languages aren’t being used for things.
                                                                                                      Go is quite popular in certain spaces too, for what it’s worth.

                                                                                                  1. 96

                                                                                                    I’m sorry to bring this up, and it’s probably considered off-topic here on Lobsters, so feel free to flag this.

                                                                                                    I know that OpenBSD and SQLite and lots of great pieces of software have been funded by the US military, and computing and military have a long and complicated relationship, but where do we as developers draw the line as to whom we are willing to accept contributions from?

                                                                                                    This is from Palantir, the company providing the technology for Trump’s deportation machine. I don’t think that this is a black/white issue, and I guess it may be possible to work at a seedy company and still do good stuff. But the docs include a FlightSearch example; is that really appropriate given the context?

                                                                                                    Regardless, thanks for releasing this as free software.

                                                                                                    1. 37

                                                                                                      Thank you very much for saying it. I think making sure these ethical topics aren’t ignored is the very least we all have a responsibility to do. It’s also entirely possible that there are people here who didn’t know about it, so it’s always worth saying.

                                                                                                      1. 37

                                                                                                        Thank you for saying this. I’m troubled by the cavalier attitude of techies toward ethics lately, and it’s nice to know I’m not alone.

                                                                                                        1. 23

                                                                                                          I don’t think a forum where this response is off-topic is worth participating in. The tech industry spends too little time thinking about the ethical implications of it’s products.

                                                                                                          1. 25

                                                                                                            Even today, we debate the ethics of using the data gathered from unethical experiments in WW2.

                                                                                                            I agree that there is a massive ethical issue working for Palatir - and I am not sure it’s ethical to use the work they have produced. Particularly if it’s a Swagger-like clone not yielding substantive value to humanity.

                                                                                                            1. 10

                                                                                                              While we’re at it, you probably typed that on a machine made by highly-exploited workers in a corrupt country that does far worse, added to the lake in the process, probably paid a surveillance-enabling company in a police state to send it over the network, and possibly wearing clothes made by kids in a sweatshop. And you did all this to get online suggesting moral folks maybe shouldn’t contribute to a HTTP/JSON thing that’s open source since a bad company might misuse [more] open source. Seems hypocritical to me.

                                                                                                              Where to we draw the line on how our consumption and contribution harms or helps others? And do you regularly do that for every product and service you buy? Most of them? Have you been active in government on laws, treaties, court cases, etc? The stuff that stops things like you describe. Or just some quick, social signaling on Lobsters getting feel-good points? If you care, I encourage you to put time into legal reform or bootstrapping alternatives to each of the things I mentioned. Maybe make for better opportunities for immigrants in whatever your country is, too. Maybe host some coding bootcamps or something for those in the slums. What you’re doing here is adding to the noise but not helping Trump’s victims or your country’s immigrants in any way.

                                                                                                              1. 71

                                                                                                                I feel like this is a great example of whataboutism.

                                                                                                                I think that if this approach was applied to tech, we’d never fix a bug because “what about the other bugs that could crash the app, this is just virtue signaling because physical compromise means game over”. Why fix a bug when you can say “What about the terrible state of security education in general, why fix a security bug when developers are just adding more?”

                                                                                                                It’s ok to make a judgement call and improve one thing in this messy world. It’s ok to try and reduce your footprint/total harm while hypocritically still participating in the system that feeds you. In fact that’s sort of core to improving those systems in a democracy.

                                                                                                                Sorry if I misinterpreted your statement, I greatly enjoy your comments across the internet.

                                                                                                                1. 11

                                                                                                                  Whataboutism is a common reply on HN or Lobsters when a popular group decries their outgroup’s activities, third party points out their actions are contrary to their own beliefs, adds that the biases indicate they’re scoring political points rather than really care, and someone pops in to say third party is whataboutism to silence those views. Thing is, whatever 3rd party brings up is almost never on these forums, getting crowd support, or whatever. Always absent. Rather than likely-intended purpose, the whataboutism claim just reinforces specific types of people supporting/rejecting specific activities by silencing dissenters. I mean, if commenter really cares about Trump’s horrors or not contributing to evil organizations, why the hell are they funding evil, slaving companies to buy toys to spend so much time on the programming projects? So, they probably don’t care or are acting like it now. Then, I do to them as they do to others.

                                                                                                                  Far as what I’m doing, I’ll tell you straight up. There’s been an increase over time of political comments that are about shaming people into behaving certain ways for a perceived, social good. Almost all of them are coming from hypocrits and/or slactivists. I mean, they’re talking on a forum no politician reads with low views. It’s not going to change Palantir’s or Trump’s practices. They know they avoiding stuff that can get results to spend time on Internet forums. So, they’re just getting an emotional high off attacking their opponents, looking like they’re responsible, or getting meaningless votes from people that agree with them. They also tie up our threads with that shit. So, as a real activist doing real-world work, I just call out their selfish, hypocritical bullshit to (a) deter more comments like that here and/or (b) encourage them to actually work on the causes they claim to work on.

                                                                                                                  Disclaimer: In fairness, people could (and do) call me out for not putting more time into actually building and deploying secure goods rather than high-level designs posted online. Although I defended my choice, I’m probably guilty of screwing up on a reasonable ratio between the two. Anything above zero code might be better. I plan to work on that more next year after I change circumstances.

                                                                                                                  Disclaimer 2: I say “almost all” cuz a few people here are legit activists or doing things at a loss to address the causes they’re talking about. I respect them a lot.

                                                                                                                  “It’s ok to make a judgement call and improve one thing in this messy world. It’s ok to try and reduce your footprint/total harm while hypocritically still participating in the system that feeds you. “

                                                                                                                  I totally agree with you. That’s not what the person was doing, though. It won’t stop Palantir’s contracts, it won’t stop the government’s activities, and proliferation of HTTP/JSON libraries will continue. The latter will even be FOSS so anyone, including Palantir, can use them. Maybe person complaining should start an alternative to Palantir that’s more ethical, organize boycotts of their products, get in a HR office poaching all their smartest talent (or delivering idiots), make enough money to pay off politicians to change government policies, and so on. Stuff that actually affects Palantir or Trump’s agencies.

                                                                                                                  “I greatly enjoy your comments across the internet.”

                                                                                                                  Thanks and same to you. :)

                                                                                                                  1. 25

                                                                                                                    Maybe person complaining should start an alternative to Palantir that’s more ethical, organize boycotts of their products, get in a HR office poaching all their smartest talent (or delivering idiots), make enough money to pay off politicians to change government policies, and so on.

                                                                                                                    This objection is absurd on its face. You can’t ethically compete in a market for unethical services. An ethical alternative to Palantir is an oxymoron, because Palantir’s ethical issues are fundamental to the things that Palantir sells. You also can’t “organize a boycott” of a defense contractor. Your final two points are literally “just have enough money to fix the problem”.

                                                                                                                    How does starting a company which sells the same thing as Palantir to the same customers Palantir sells to, hires the same people as Palantir, has the same wealth as Palantir, and bribes politicians the way Palantir does, stop the problem of companies that behave like Palantir? You’re objecting to someone criticizing the status quo by telling them they should instead… further reinforce the status quo?

                                                                                                                    1. 19

                                                                                                                      I think you misapprehend what is going on here. This is a forum for highly technical people; by raising the serious ethical space Palantir exists in, it directly bears on creating difficulty in recruiting, along with decreasing retention.

                                                                                                                      You, of all people, should understand the power of words on an internet screen to influence readers: you’ve been writing long & grammatically correct essays on security across multiple major internet fora for years. I’ve seen you on Schnier and HN, :) Communication, persuasion, and discussion are an essential activist activity. (And for my money, it is substantially more effective than picketing and marching 95% of the time…)

                                                                                                                      1. 7

                                                                                                                        (I suspect this was meant as a reply to the person I replied to.)

                                                                                                                        1. 2

                                                                                                                          “by raising the serious ethical space Palantir exists in, it directly bears on creating difficulty in recruiting, along with decreasing retention.”

                                                                                                                          I agree with you. I actively do that in real life every day for customers and coworkers wanting something better in a lot of areas. I have plenty of results to show for it. That’s because I put the time in where it gets results and consistently do it rather than one-off’s we sometimes see here. Companies like Palantir use recruiting practices that cast a wide net. Anyone wanting to disrupt their recruiting should be posting such comments on sites with massive numbers of page views that are mostly developers. Big, social media sites like Twitter, Facebook, Reddit, and Hacker News. LinkedIn, too, if you can do it that way but I haven’t been on in long time. That’s why I encourage them to put political efforts in front of literally millions of developers instead of a hundred or less participating here if aiming for a big wave of change.

                                                                                                                          “You, of all people, should understand the power of words on an internet screen to influence readers: you’ve been writing long & grammatically correct essays on security across multiple major internet fora for years. I’ve seen you on Schnier and HN, :) “

                                                                                                                          You said long and grammatically correct. You gotta be messing with me on second half lmao. I agree with the power of words and persuasion as stated above. Hell, you had to have seen me do it there, esp to “Skeptical” (troll or near-perfect DOD apologist) before I left. That’s why I tell them to use that power where it gets results instead of Lobsters. Then, we keep Lobsters focused on deep, technical stuff with low noise. Anyone wanting to achieve political action can ping Lobsters, via dedicated threads or private messages, to go where the action is to get actual, Palantir-scale results.

                                                                                                                          ““It is what it is”, which is what your comment & Nick’s comment promote, simply promotes apathy; history provides many examples of change taking place. I encourage people to shake off the belief that things will always stay the same.”

                                                                                                                          That’s not true at all. I’ve just followed something like several lifetimes worth of history on the U.S. military and government under both left- and right-leaning leaders finding the military-industrial-complex just got more powerful over time. The politicians of both sides support it. The right supports companies like Palantir overtly. The left’s politicians will support the defense contractors for both payouts and to bring jobs to their districts. So, to change the situation voronoipotato describes, you have to get millions of people to vote out scumbags that take money to improve chances of elections to combat defense industry or get an anti-war, pro-immigration President in office with Congress willing to roll-back legislation.

                                                                                                                          The last election surprised most lefter-than-I liberals that were trying to make people say the correct things on forums, etc in ways we see in some threads here. I doubt they’re capable of achieving that 180 directly if keeping same practices that failed before so hard they didn’t even see what was coming. Fingers crossed that we just get lucky that Trump does so much damage and embarrassment that a reversal happens in swing states after the Democrats get on top of their shit this time. Or we centrists get a President. Fat chance on that one since few listen to moderates. ;)

                                                                                                                        2. 5

                                                                                                                          The person you’re talking to likely doesn’t even think that Defense Contracting is unethical. Being said palantir is going to keep existing, boycotting doesn’t mean anything here because we don’t even buy their products. Even under a proper organized effort if we got a different defense contractor absolutely nothing would be different. The only tactics I’m aware we can do are mitigation tactics of not giving our labor to defense contractors, but this drives up the wages to the point where someone would. You can if you work there do a labor slowdown, but your ability to act in that way is limited, and useless if it’s not a group effort.

                                                                                                                          Palantir is a bad thing but our ability to affect it is extremely limited. Electoral politics is mostly useless here. Their lobbying power affects both parties pretty evenly. IMHO it’s better to put energy into mitigation tactics into problems where it’s easier to have traction. One group has been for example paying for bail bonds for refugees.

                                                                                                                          Defense contractor spending isn’t a symptom of capitalism but rather attached to the heart, a swollen vestigial organ from mercantilism and much like the appendix may kill you if you remove it unskillfully.

                                                                                                                          I think it’s natural to see the biggest problem and try and lock horns with it, but sometimes a smaller problem you can solve is genuinely better than a larger problem you can’t. Obviously don’t work for them, there’s plenty of other places that pay you well and you won’t even have to think about all the bodies when you go to sleep.

                                                                                                                          1. 7

                                                                                                                            The person you’re talking to likely doesn’t even think that Defense Contracting is unethical.

                                                                                                                            Yes, but the person they’re suggesting this in response to does, which was the context of nickpsecurity’s original suggestion to compete with Palantir.

                                                                                                                            The only tactics I’m aware we can do are mitigation tactics of not giving our labor to defense contractors, but this drives up the wages to the point where someone would.

                                                                                                                            I don’t know what your point is. Driving up wage costs for unethical corporations is the point of organizing an effort to boycott employment at specific corporations. The goal is making things like human rights violations untenable to corporations by making them unprofitable. Yes, this is a half measure - but it’s not nothing, either.

                                                                                                                            Defense contractor spending isn’t a symptom of capitalism but rather attached to the heart, a swollen vestigial organ from mercantilism and much like the appendix may kill you if you remove it unskillfully.

                                                                                                                            So your point is, we should leave it alone?

                                                                                                                            I think it’s natural to see the biggest problem and try and lock horns with it, but sometimes a smaller problem you can solve is genuinely better than a larger problem you can’t.

                                                                                                                            On the contrary - refusing to work for companies like Palantir and encouraging my fellow tech workers to do the same is one of my most fruitful opportunities to fight against systemic injustices at the moment. Each of us in the tech industry have far more influence on an our industry’s actions than on the actions of things like the federal government - there are less than four million programmers in the entire US, as opposed to the vastly higher number of voters. We should be adamant about using our privileged place as one of the few labor pools left with real negotiating power to prevent our industry from committing acts of evil, not conveniently defeatist whenever someone dares to suggest the small personal sacrifice of choosing not to directly build the tools of human misery.

                                                                                                                            1. 7

                                                                                                                              Fundamental changes are achieved by many people choosing to not accept what is, and coming together to push towards a major change in the status quo.

                                                                                                                              “It is what it is”, which is what your comment & Nick’s comment promote, simply promotes apathy; history provides many examples of change taking place. I encourage people to shake off the belief that things will always stay the same.

                                                                                                                              1. 1

                                                                                                                                You said it even better than me.

                                                                                                                            2. 20

                                                                                                                              Whataboutism is a common reply on HN or Lobsters when a popular group decries their outgroup’s activities, third party points out their actions are contrary to their own beliefs, adds that the biases indicate they’re scoring political points rather than really care, and someone pops in to say third party is whataboutism to silence those views. Thing is, whatever 3rd party brings up is almost never on these forums, getting crowd support, or whatever.

                                                                                                                              No it’s a common reply when you distract from the discussion at hand to go ‘oh but what about these other unrelated issues?’ Your response is literally at the level of ‘capitalism made your iPhone you’re using to have this conversation so checkmate’ in a discussion about economic systems.

                                                                                                                              There is no ‘popular group’ here, there’s no ‘outgroup’, nobody is decrying anyone’s activities. You haven’t ‘pointed out’ any actions that are contrary to anyone’s beliefs or exposed any biases or virtue signalling. All you’ve done is responded to a post pointing out that Palantir might be an unethical company, accusing them of virtue signalling! They didn’t even say ‘Palantir is bad’. They suggested that it might be, and that it was worth thinking about and discussion. Did you then discuss it? Did you think about it? No, you just launched into an attack, said that their post was social signalling and accused them of hypocrisy.

                                                                                                                              Imagine for a moment the discussion was oil companies, and the person you were responding to had said ‘I think oil companies often act unethically and I think we should consider whether we want to be working with them and contributing to their open source software’. Your response was the equivalent of ‘you don’t have an electric car so you’re not allowed to discuss this’. I hope you can see that that is nonsense.

                                                                                                                              I totally agree with you. That’s not what the person was doing, though. It won’t stop Palantir’s contracts, it won’t stop the government’s activities, and proliferation of HTTP/JSON libraries will continue. The latter will even be FOSS so anyone, including Palantir, can use them. Maybe person complaining should start an alternative to Palantir that’s more ethical, organize boycotts of their products, get in a HR office poaching all their smartest talent (or delivering idiots), make enough money to pay off politicians to change government policies, and so on. Stuff that actually affects Palantir or Trump’s agencies.

                                                                                                                              When someone says ‘where do we as developers draw the line as to whom we are willing to accept contributions from?’ they are opening up a discussion. Maybe the result of that discussion would have been ‘anyone actually’. Suggesting that the first thing you should do is start boycotting companies before the issue has even been discussed is ridiculous. Discussions are fine. Discussions are not slacktivism. Posting ‘#stoppalantir #metoo #stoptrump’ at the end of your tweets and doing nothing else in your life is virtue signalling. Discussing issues is not.

                                                                                                                              1. 10

                                                                                                                                There is no ‘popular group’ here, there’s no ‘outgroup’, nobody is decrying anyone’s activities.

                                                                                                                                A person submitted a HTTP/JSON toolchain that they were open-sourcing. A versatile, general-purpose tool that can be used for good if someone wants to. The comment I replied to ignored the software submission entirely to tell them they’re unethical for working at Palantir since other parts of the company uses its tech to serve an unethical customer. That’s decrying activities. Such reasoning also applies to companies like Google (or other surveillance companies), Apple/Foxconn, VC-funded companies aiming for lock-in, and so on since buying their stuff or contributing to their FOSS might support all kinds of evil. Some people supporting the decrying comment even work at such companies despite other jobs being available for people with that kind of talent. Strange.

                                                                                                                                The fact that this accusation and suggestion to quit their job got 60 votes vs 7 about the submission… on Lobsters with lower numbers of votes to begin with… definitely says it’s popular. The marked difference between the people who support or question that tangent supports the existence of an outgroup relationship. I can’t say as much about what it means here since the outgroup receives more support on a lot of political divides. Lots of folks here hate companies like Palantir regardless of other beliefs. That’s what I’m leaning toward.

                                                                                                                                It’s been an interesting thread to observe, though.

                                                                                                                                1. 2

                                                                                                                                  Wholeheartedly agree, there! I suspect I drew different conclusions than you, though.

                                                                                                                              2. 3

                                                                                                                                People can disagree with you without being part of a conspiracy to silence or shame you. Maybe a less emotional response would be more informative.

                                                                                                                              3. 0

                                                                                                                                One of nick’s pastimes seems to be railing against liberal “hypocrisy” on this website, mostly by deflecting into muddy tangential arguments just like so.

                                                                                                                                1. 13

                                                                                                                                  Please don’t post ad-hominem attacks here. If you disagree with the argument, pick it apart politely.

                                                                                                                                  Lord knows you should have enough practice by now to do so.

                                                                                                                                  1. 5

                                                                                                                                    If you disagree with the argument, pick it apart politely.

                                                                                                                                    That only works if both sides are arguing in good faith though which definitely doesn’t appear to be the case with some commenters on here.

                                                                                                                                    1. 4

                                                                                                                                      If that’s the case, then arguing further with somebody in bad faith is just going to create noise and antagonize other lobsters. Best just to ignore the posts then.

                                                                                                                                      1. 3

                                                                                                                                        I do but it ruins the lobsters experience for me to see people arguing in bad faith without any censure. Some of them even seem to be encouraged as a kind of clickbait/outrage generator. It’s disheartening.

                                                                                                                                    2. 4

                                                                                                                                      Lord knows you should have enough practice by now to do so.

                                                                                                                                      This is an ad-hominem, friendly.

                                                                                                                                2. 19

                                                                                                                                  Leaving whataboutism aside, I think you cannot conflate the (delusional) idea of ethical consumption with active usage and contribution of open source software.

                                                                                                                                  Ethical consumption doesn’t work for the structure of the market, where the contribution of the individual gives no perceivable feedback to the system.

                                                                                                                                  The Open Source world and software engineering are a much smaller world. It is a realistic goal to radicalize enough software engineers inside and outside of Palantir in order to halt their production. Your target audience has contract leverage, money and is highly connected and easily reachable.

                                                                                                                                  This is a much easier and realistic goal than convince the management of some big corporation to reduce their exploitation just because a small minority of consumers is unhappy. When they realize this, instead of reducing exploitation, they invest in more marketing to wash their brand, or they simply start a new one. Much cheaper.

                                                                                                                                  Don’t conflate your power as a consumer with your power as a producer, because they very different.

                                                                                                                                  1. 11

                                                                                                                                    I used to work for Nokia. They did everything in their power to ethically source all their materials. It was the only phone company that did that. Other companies don’t do that because nobody demands it from them. While there is no ethical consumption under capitalism, there is slightly less terrible consumption. So where do we draw the line? As deep into their pocket books as it can go.

                                                                                                                                    1. 1

                                                                                                                                      I didn’t know that about Nokia. That’s awesome! Thanks for the tip.

                                                                                                                                      1. 1

                                                                                                                                        Now, keep in mind the new Nokia phones are made by a different company that just licenses the brand. I’m not sure if care as much.

                                                                                                                                    2. 10

                                                                                                                                      […] the lake […]

                                                                                                                                      That is horrible.

                                                                                                                                      Seems hypocritical to me.

                                                                                                                                      Ok.

                                                                                                                                      Where would you draw the line personally? Do I understand your opinion correctly as suggesting that if you use a computer, then you shouldn’t be discussing unethical behaviour, e.g. racism? It is not my intention to judge here; just genuinely curious.

                                                                                                                                      Maybe make for better opportunities for immigrants in whatever your country is, too.

                                                                                                                                      I agree with this very much, and this is something that I aspire to do. Additionally I do have friends that have been deported, and worry a bit about my own not so distant post-Brexit situation in the UK.

                                                                                                                                      1. 2

                                                                                                                                        Im glad you’re doing real work on this issue. I commend that.

                                                                                                                                        Writing it here likely isn’t is the thrust of my point. Instead, it’s just adding noise to the forum plus sending a jab at one of only folks we know in Palantir doing something possibly beneficial (eg open-sourcing software for data analysis). The people here that would agree with your position already dont work for Palantir, use their services, or vote for folks that support horrible policies on immigration.

                                                                                                                                        Those that do these thing are (a) mostly not on Lobsters where your comments bave about lowest ROI you can get and (b) usually disagree with you with no intent to change their mind based on your comment that states the obvious. So, you’re not reaching them. Goes for similar comments aiming for political impact on government-level stuff in non-political, Lobsters threads. So, I push for people not to introduce them.

                                                                                                                                        Im at work now so responses might be delayed.

                                                                                                                                        1. 5

                                                                                                                                          mostly not on Lobsters where your comments bave about lowest ROI you can get

                                                                                                                                          Yes, you are probably correct in that observation.

                                                                                                                                          I wasn’t really sharing my thoughts here expecting any impact, but rather because I’m interested in hearing what other people think. And you are right that I’m being hypocritical here, because I doubt I’d react the same to an IBM project even though they have a shameful past; and even worse, I used to work on this phone app promoting some agrochem from DOW. At first I just kept my eyes on the code, but I couldn’t justify it to myself after reading about their role in the Vietnam War and the Bhopal Disaster and all that.

                                                                                                                                          So, it was intended more of an open question about where people here draw the line.

                                                                                                                                          1. 2

                                                                                                                                            Well, you seem to be speaking out of the heart on things you’ve been close to personally. I respect that. I still say low-ROI with better results elsewhere. You were bringing it up for good reasons, though. The community response also strongly favored your comment in a way consistent with prior threads on politics showing a shift in what Lobsters wants as a community. I’ll write on that in the future.

                                                                                                                                            And it’s still cool you’re another person reusing older computers with the LiveCD tests and such. Off-topic a bit, but I was wondering if the hardware vulnerabilities they probably won’t patch on 5-10 year old machines have you considering new stuff? I always knew they were there. Now, they’re coming quickly with many eyeballs on them. Might be best reason I ever heard to get the latest and greatest from Purism, Raptor, or whoever. And then most have backdoors for (insert group) but fewer hardware 0-days for (more groups). Wait, I thought this tangent-tangent could lighten things up with easier choices… Looks just as hard… ;)

                                                                                                                                            1. 1

                                                                                                                                              Off-topic a bit, but I was wondering if the hardware vulnerabilities they probably won’t patch on 5-10 year old machines have you considering new stuff?

                                                                                                                                              I don’t know enough about this; what hardware vulns are we talking about here, and how easy are they to exploit? Although it’s not really about hardware, there’s that whole Intel Management Engine issue that is avoided by using somewhat old WinXP-era 32-bit laptops, so newer is not always more secure.

                                                                                                                                              And it’s still cool you’re another person reusing older computers with the LiveCD tests and such.

                                                                                                                                              Oh yes that thread! At least it’s a bit less harmful if we can use computers for longer. A friend of mine has a Mac that can’t get more OS X updates now, so she’s stuck with insecure versions of Firefox and all that. Gonna put Debian on it later this week, hopefully!

                                                                                                                                              Do you know of any somewhat more ethical laptop producers?

                                                                                                                                              1. 2

                                                                                                                                                re hardware attacks.

                                                                                                                                                Essentially, the hardware has always been secure with only elite pro’s focusing on it. Now, due to Meltdown/Spectre, hardware attacks have gone really mainstream with all kinds of university research, private research, folks describing stuff on blogs, and so on. All the CPU’s that were highly optimized (esp Intel/AMD) are vulnerable to them needing patches. They’re doing the attacks over the network now. Older systems used to be safer but now they’re not since attacks will get more numerous and effective over time.

                                                                                                                                                About the only things that are immune were simpler, embedded CPU’s. They’re not designed for security, though, with far less attention by defenders. So, that could reduce the hardware risk adding to the software risk. Simpler boards that can run modern, security-updated OS’s might help. I’m not sure. At lot of stuff is piling in.

                                                                                                                                                re put Debian on it.

                                                                                                                                                Ok, you’re already using that strategy. Good thinking and being helpful! :)

                                                                                                                                                re ethical producers

                                                                                                                                                I can’t remember since I was buying used ones like you. The one I see in media most with good things going for it is Purism. They try to disable the ME with software changes, too. Some folks pushing high freedom were using MiniFree to get ME-less, FOSS-firmware laptops. It had downsides. My own Core Duo 2 still handles stuff nicely outside games, highest-def content, and worst of web apps. Here’s a Guardian article I just found with some recommendations. Another said iFixit can help keep things going.

                                                                                                                                                So, not a lot of options for new hardware minimizing harm to self and others. There are options in both reuse and new categories that help us reduce harm. We can at least do that. I stay dedicating slices of my research to solving this problem. Tracking whatever can help for whoever can do it. Maybe something will shake out eventually.

                                                                                                                                        2. 0

                                                                                                                                          Additionally I do have friends that have been deported

                                                                                                                                          Sorry but are we now living in a world where the ‘standard’ left-wing political view in the Anglosphere is that any kind of deportation is bad? Because that’s how I’m reading this comment.

                                                                                                                                          Immigration policy exists for very good reasons. The American political dichotomy that either there should be zero immigration or completely unchecked immigration is, for lack of a better word, moronic.

                                                                                                                                          1. 3

                                                                                                                                            I think it’s fair to assume that the poster could be criticising the particular immigration policy that led to these deportations, instead of all immigration policy.

                                                                                                                                            1. 1

                                                                                                                                              It could be fair, if the poster denounced similar and almost identical policies under the previous President. As it stands, the poster is mostly just criticizing immigration policies that seemed totally reasonable and main stream just eight short years ago.

                                                                                                                                        3. 5

                                                                                                                                          You can’t make perfect the enemy of good. Your argument essentially seems to be that if you can’t live perfectly you shouldn’t try living better at all.

                                                                                                                                          It’s virtually impossible to operate in the modern world without using the internet, without having and using a computer. If it were possible to, for a reasonable price that I can afford but knowing I’d have to pay somewhat of a premium, buy a computer that I knew wasn’t made using exploitation of those in the third world, then of course I would buy one. But I don’t know that it is. And there are other competing priorities too, like getting a computer that is free of binary blobs and proprietary software.

                                                                                                                                          I certainly don’t pay a ‘surveillance-enabling company in a police state’ to send anything over the internet. I pay an ISP for internet access, but I don’t live in a police state and as far as I know my ISP doesn’t enable surveillance.

                                                                                                                                          In the same way that I think it’s perfectly reasonable for someone to say ‘I can’t afford to be vegan’ even though being vegan is morally important, I think it’s perfectly acceptable to say ‘I can’t afford to buy ethically produced clothes’. Plus there’s significant evidence that manufacturing things in third world countries has improving their living standards and wages considerably.

                                                                                                                                          Where to we draw the line on how our consumption and contribution harms or helps others? And do you regularly do that for every product and service you buy? Most of them?

                                                                                                                                          I like to have an idea, at least, of what goes into the things I buy, yes. It’s hard to do it with absolutely everything though, because there’s just so much different stuff.

                                                                                                                                          Have you been active in government on laws, treaties, court cases, etc? The stuff that stops things like you describe.

                                                                                                                                          That’s absolutely ridiculous. You do not have to be a member of government to have a political view. You do not have to negotiate an international treaty to have a political view. You do not have to sue someone to have a political view. Your standards are ridiculous.

                                                                                                                                          Or just some quick, social signaling on Lobsters getting feel-good points?

                                                                                                                                          Discussing important issues is not ‘virtue signalling’.

                                                                                                                                          If you care, I encourage you to put time into legal reform or bootstrapping alternatives to each of the things I mentioned. Maybe make for better opportunities for immigrants in whatever your country is, too. Maybe host some coding bootcamps or something for those in the slums. What you’re doing here is adding to the noise but not helping Trump’s victims or your country’s immigrants in any way.

                                                                                                                                          This has nothing to do with immigrants and everything to do with Palantir being a company that operates in an unethical manner. It’s a surveillance company. There’s absolutely nothing problematic about a company producing software on contract for a government that has immigration policies. The issue is that Trump’s policies are violating human rights in how they’re enforcing those laws.

                                                                                                                                          You don’t solve this problem by creating ‘coding bootcamps’ for immigrants LOL.

                                                                                                                                        4. 4

                                                                                                                                          I guess it may be possible to work at a seedy company and still do good stuff.

                                                                                                                                          Regardless, thanks for releasing this as free software.

                                                                                                                                          Every field of endeavor is welcome here. Every field of endeavor is welcome here for technical discussion, free of (without expectation of) moralizing, guilt, or shame.

                                                                                                                                          1. 2

                                                                                                                                            I personally already draw the line at technology coming from uber for ethical reasons, so I will not touch palantir things at all. Thanks for bringing that up!

                                                                                                                                          1. 14

                                                                                                                                            I’m all for unionizing, not out of “unionize all the things”, but because I’m interested in how different power structures might effect different results: experimentation, if you will, but not at the cost of free association.

                                                                                                                                            I grew up in a family of public school teachers. I’ve seen unions mess up badly.

                                                                                                                                            1. 24

                                                                                                                                              My wife is a teacher in the public system and we both have mixed feelings about the union representing teachers. It does some wonderful things, but then goes out of its way to defend some absolute garbage people simply because they are in the union. And heaven forbid it if you have any critcisms of the how the union operates. Neither of us think it’s so much a union thing as it is the lack of care put into building a large organization since similar problems exist in companies.

                                                                                                                                              1. 14

                                                                                                                                                I’m not super-attached to the idea of unions, but it’s pretty obvious to me that we are getting exploited by the companies–especially startups–that we work for.

                                                                                                                                                I’m not sure that a full-blown union system is the answer, mostly because I trust the soft skills and systems thinking of engineers about as far as I can thrown them, but we need to start organizing as a class of labor on some basic things that keep screwing up the market for all of us:

                                                                                                                                                • Forced arbitration
                                                                                                                                                • Broad NDAs
                                                                                                                                                • Broad non-competes
                                                                                                                                                • Broad assignments of invention and other IP
                                                                                                                                                • Lack of profit sharing
                                                                                                                                                • Bad equity for early-mid stage engineers
                                                                                                                                                • Uneven salary systems

                                                                                                                                                Every company and startup gets some of these wrong, and few (if any of them) right, but because it’s accepted as “standard practice” we all end up having to endure them.

                                                                                                                                                I don’t think we can find a one-size-fits-all solution for, say, salary ranges or other more esoteric issues, but my belief is that those specific things enumerated above are both achievable and universally beneficial for developers. They would benefit both the folks that think they can be the smartest engineering in the company and somehow make out like in the 90s, and the lifers who just quietly and competently do their jobs and switch companies when it’s time.

                                                                                                                                                We need to push for them.

                                                                                                                                                1. 2

                                                                                                                                                  “I trust the soft skills and systems thinking of engineers about as far as I can thrown them”

                                                                                                                                                  I was a bit surprised to read that. I know engineers are infamous for falling short on “soft” skills but isn’t systems thinking supposed to be a forte of engineers?

                                                                                                                                                  1. 2

                                                                                                                                                    One would think so!

                                                                                                                                                    In my experience the first thing most smart (note: not wise, just smart) engineers reach for when consulted with a misbehaving situation, especially involving humans, is a system. They have this idea that some intricate set of deterministic protocols and social customs will save them from the ickiness and uncertainty of dealing with other sentient rotting meat. They’re invariably wrong.

                                                                                                                                                    Outside of dealing with other people in meatspace, my current work in web stuff has similarly colored my opinion of “systems thinking”, to the point where I basically don’t trust anybody to reliably engineer anything larger than a GET route backed by a non-parameterized query to a sqlite database–they tend to want to add extra flexibility, containers, config files, a few ansible scripts for good measure, maybe some transpiler to the mix to support a pet stage 1 language feature, and all this other nonsense.

                                                                                                                                                    So, sadly, I’m reluctant to trust those folks who overengineer and underempathize to successfully build and manage a union.

                                                                                                                                                    1. 2

                                                                                                                                                      Engineers are famous for thinking that a new bit of technology could revolutionize systems which include human social behaviors.

                                                                                                                                                      I’ve met 2-3 engineers in the past decade who I would call ‘systems thinkers’. I’d like to make it onto my own list, someday.

                                                                                                                                                  2. 6

                                                                                                                                                    I have on my reading list https://www.press.uillinois.edu/books/catalog/47czc6ch9780252022432.html, which talks about the self-organized unions in 1930s that preceded NLRB, and the ways in which they were more democratic and more responsive to membership.

                                                                                                                                                    1. 1

                                                                                                                                                      I eagerly await your synopsis of it and maybe I’ll pick it up myself. I enjoy your writing!

                                                                                                                                                      1. 0

                                                                                                                                                        Taft-Hartley in the 1950s had a terrible effect on unions, partly by banning wild-cat strikes and boycotts both of which forced union leaders to be responsive to members.

                                                                                                                                                    1. 4

                                                                                                                                                      At least US immigration is protecting us from Curl [sarcasm indicator]

                                                                                                                                                      1. -1

                                                                                                                                                        too close to the cuff

                                                                                                                                                      1. 7

                                                                                                                                                        This is quite a stretch.

                                                                                                                                                        Why is it that tech workers don’t think of themselves as “workers”?

                                                                                                                                                        Why is it that governmenty-people want everyone to think of themselves as a worker? It would benefit everyone if we moved towards more autonomy, not less. The “employee” status is a crutch, and trap, and way of giving bureaucrats more control. Why isn’t everyone their own corporation? More sovereignty, not less.

                                                                                                                                                        called out for colluding via anti-poaching schemes to suppress wages. As part of a longer-term strategy, these tech giants have also partnered with schools, supported coding bootcamps, and sponsored programs that teach minority youth groups to code.

                                                                                                                                                        Haha, that’s a hot potato. I’m guessing that message will need a massage before it reaches the union talking points slidedeck.

                                                                                                                                                        Their goal in technology education is not to simply expand their consumer base to the youngest and most marginalized of our society, but to increase the labor supply of the future and subsequently drive down wages for decades to come.

                                                                                                                                                        Unions increase friction. Friction reduces mobility. No, thanks.

                                                                                                                                                        It’s hilarious to me that we have the privilege of existing in a time where control freaks haven’t yet figured out a way to regulate this thing called software, and because of that, a very fluid and egalitarian market exists where you can go from high school, prove yourself on a whiteboard, and get a pretty nice job.

                                                                                                                                                        And yet the Local 41 people want to bring in the overalls and flannel and make this industry the same as all of the other industries that have been ruined. And they complain about whiteboard interviews, and want to replace it with committees and certifications and degrees.

                                                                                                                                                        Be careful about the people with whom you raise pitchforks. Or don’t, I don’t care. But I told you so.

                                                                                                                                                        1. 22

                                                                                                                                                          Why is it that governmenty-people want everyone to think of themselves as a worker? It would benefit everyone if we moved towards more autonomy, not less. The “employee” status is a crutch, and trap, and way of giving bureaucrats more control. Why isn’t everyone their own corporation? More sovereignty, not less.

                                                                                                                                                          There’s a difference with diagnosing how things are and how things should be.

                                                                                                                                                          Anti-poaching agreements between the top tech companies is a thing that happened. Everyone in SF had their wages surpressed because the owners of these huge places decided they shouldn’t have to fight in a fair labor market. It is undeniable, there is court evidence and everything, that there was massive collusion to make sure that those who had the shortsighted-ness to go into an employment agreement with them would have their mobility reduced.

                                                                                                                                                          Yeah, sure. Would be great if people could just work and have relations of trust. But so long as the person on the other side of the negotiating table has massive infrastructure dedicated to paying the least amount of money to you, standing alone is a sure-fire way to get a worse deal.

                                                                                                                                                          There’s tribalism and other organizational ills in unions, but…. welcome to “put more than 3 people in a room”. There’s self-interest and politics in every org.

                                                                                                                                                          The egalitarian software market creates something where people like Lewandowski can go around and collect millions off of claiming other people’s work as his own, and other people can’t even get interviews at other companies because of secret agreements. It’s a market where the loudest people get the biggest wins (see every single post about organizational dysfunction at Google that means that nothing actually gets maintained).

                                                                                                                                                          1. 3

                                                                                                                                                            The egalitarian software market creates something where people like Lewandowski can go around … where the loudest people get the biggest wins …

                                                                                                                                                            Assuming unions and regulations solve that somehow, it’s a petty reason to install an long-term bureaucracy. What did we gain exactly, besides sticking it to some (imagined) character you didn’t like?

                                                                                                                                                            1. 2

                                                                                                                                                              “other people can’t even get interviews at other companies because of secret agreements”

                                                                                                                                                              That’s not what the secret agreements were about; they agreed not to cold-call each others employees to try and get them to leave but they did not agree to not interview each others employees if they applied to positions.

                                                                                                                                                            2. 6

                                                                                                                                                              It would benefit everyone if we moved towards more autonomy, not less

                                                                                                                                                              Autonomy is (rightly) coupled to responsibility for outcomes.

                                                                                                                                                              Employees are not given autonomy or responsibility for outcomes (for so, so many reasons).

                                                                                                                                                              Autonomy is worthless unless you can capture the benefits of doing a good job.

                                                                                                                                                              1. 1

                                                                                                                                                                Unions increase friction. Friction reduces mobility. No, thanks.

                                                                                                                                                                It’s hilarious to me that we have the privilege of existing in a time where control freaks haven’t yet figured out a way to regulate this thing called software, and because of that, a very fluid and egalitarian market exists where you can go from high school, prove yourself on a whiteboard, and get a pretty nice job.

                                                                                                                                                                And yet the Local 41 people want to bring in the overalls and flannel and make this industry the same as all of the other industries that have been ruined. And they complain about whiteboard interviews, and want to replace it with committees and certifications and degrees.

                                                                                                                                                                Be careful about the people with whom you raise pitchforks. Or don’t, I don’t care. But I told you so.

                                                                                                                                                                The assumption that wanting to regulate the software industry is the mark of “control freak” is way off. Software is no longer marginal. It has long since eaten the world. It affects every detail of most people’s lives in the first world. Anything with those implications needs regulation. It needs a class of workers who are trained in a semi-standard fashion that includes concerns about the ethical implications of what they’re doing. In other words, it needs to be an actual engineering discipline.

                                                                                                                                                                Also, this specifically seems way off base to me:

                                                                                                                                                                a very fluid and egalitarian market exists where you can go from high school, prove yourself on a whiteboard, and get a pretty nice job.

                                                                                                                                                                Is the implication there that the hiring market right now is easy to move around in, but with more standardization it wouldn’t be? That just doesn’t jive with the data that exists out there for tech hiring. It’s essentially random right now, there is literally no such thing as a meritocracy. Companies are making hiring decisions based on which way the wind blows and whether or not your interviewer liked what they ate for breakfast. More industry wide standardization would make things much better.

                                                                                                                                                                1. 4

                                                                                                                                                                  It affects every detail of most people’s lives in the first world. Anything with those implications needs regulation.

                                                                                                                                                                  I don’t know where the idea came from, but just because something is ubiquitous does not mean it needs to be regulated. Rather it happens that ubiquitous things attract control freaks because they are predisposed to the obvious.

                                                                                                                                                                  It needs a class of workers who are trained in a semi-standard fashion that includes concerns about the ethical implications of what they’re doing. In other words, it needs to be an actual engineering discipline.

                                                                                                                                                                  “It” (software) is a very generic thing, and a future where one needs to consult a trade guild to sell software, will be sad.

                                                                                                                                                                  It’s essentially random right now, there is literally no such thing as a meritocracy.

                                                                                                                                                                  Compared to other industries, I don’t see how you can make that claim. “Merit” is demonstrated ability, and it is clearly the case that many tech professionals are hired based on some demonstration of ability rather than only credentials, connections, etc.

                                                                                                                                                                2. -1

                                                                                                                                                                  I agree. This socialist “class”-based thinking, along with the fundamental economic misunderstandings of the socialist worldview is fundamentally dangerous. Besides, unions are bureaucracy and bureaucracy makes everything worse for everyone. I don’t want people to organise, especially not on my behalf. It’ll undoubtedly make my life harder. Or does this writer know what’s best for me?

                                                                                                                                                                  I’ll advocate for my own interests, thanks.

                                                                                                                                                                  1. 22

                                                                                                                                                                    I really like the bureaucracy that inspects milk for contaminents, prohibits people dumping benzene in the water, and vaccinates children. Probably I’m not a good person.

                                                                                                                                                                    1. 9

                                                                                                                                                                      It’s bizarre that as the level of union participation went down , the amount of bureaucracy went up. Also note an interesting fact, when the Soviet Union collapsed and the government industry privatized following the recommendations from Chicago school economists, the total number of government bureaucrats increased! Capitalism is really good at creating bureaucracy, both public and private.

                                                                                                                                                                      1. 6

                                                                                                                                                                        …fundamental economic misunderstandings of the socialist worldview…

                                                                                                                                                                        Besides, unions are bureaucracy and bureaucracy makes everything worse for everyone.

                                                                                                                                                                        You seem to think you know what’s best for me… This is a silly argument. No one is advocating for you any more than you are advocating for other people by advocating against socialism on the Internet.

                                                                                                                                                                        1. 8

                                                                                                                                                                          At least in Australia where I live, union political campaigns lead to:

                                                                                                                                                                          • Making employers liable for employee deaths and injuries at work (leading to a huge drop in workplace deaths and injuries)
                                                                                                                                                                          • Meal breaks (previously factory workers had a full days shift without food)
                                                                                                                                                                          • Annual and sick leave
                                                                                                                                                                          • Maternity leave
                                                                                                                                                                          • A high minimum wage (initially enough that one fulltime job could support a family; has declined since)
                                                                                                                                                                          • 15-20% more income (according to the governments Australian Bureau of Statistics)

                                                                                                                                                                          That’s not to say they don’t do any harm. Variously:

                                                                                                                                                                          • Education union rules made it impossible to fire permanent teachers, causing schools to employ tons of staff on precarious 6-months contracts.
                                                                                                                                                                          • The dockworkers union picketed after a untrained, non-union crew were brought in and did the job faster with a better safety record.
                                                                                                                                                                          • There are constant claims of criminal behavior among the CFMEU (though how much of that is true vs Murdoch is hard to say)
                                                                                                                                                                      1. 2

                                                                                                                                                                        Sounds like a call to use Mathematica.

                                                                                                                                                                        1. 5

                                                                                                                                                                          Prepping for a trip to Turkey for a week near the Syrian border to provide some training to humanitarian staff working in Syria on disease and health monitoring and outbreak management. Never been to Turkey but have friends from there, super excited to go but don’t know a lot about turkey from a cultural standpoint. Need to ask friends for some help.

                                                                                                                                                                          Trying to get the Congo set up in our disease management system as well in between playing with the kids and prepping training materials as well.

                                                                                                                                                                          1. 2

                                                                                                                                                                            Sincerely, bravo. You’re a hero.

                                                                                                                                                                            1. 1

                                                                                                                                                                              Really interesting. What do you train people do do?

                                                                                                                                                                              1. 4

                                                                                                                                                                                I’m the sole developer for a fairly large data collection and analysis system focused on disease surveillance and outbreak management. It’s funded as a consultancy to a part of the U.N. I’ve been at it for about five years now (before that I built systems for health management in refugee camps, situations, etc…). We’re set up in emergencies and as long-term surveillance in about 12 countries at the moment, mostly centered around Africa but some in the Pacific, Middle-East, etc…

                                                                                                                                                                                I usually travel a few times a year, but haven’t in the last year as i’ve been waiting for my permanent residence application to get finalized. But normally I go out to the field with an epidemiologist and do field training for field staff, IT staff, etc… and it’s just a good opportunity to sit down and get a better understanding of what limitations people are hitting with the system and how I can improve things, and people like to put a face to the name as well when they hear that it’s just this one dude that’s going to be underpinning an entire sub-section of their operation. These people are usually pretty busy so if something doesn’t work for them, they’ll abandon it pretty quickly and go back to excel or paper-based reporting… we’re on the cusp of releasing a locally installable version of our system as rust binaries and the Syrian conflict is going to be one of the first to get it and have it installed locally so need to sort of assess what they have for infrastructure, etc…

                                                                                                                                                                                The whole thing is completely free though, doesn’t cost countries/orgs a dime unless they want a “kit” (basically a bunch of pre-configured phones, couple laptops and a local relay server). It’s a funded project though (and all the merciless stress that comes with that) I actually paid for the application servers for some of the largest outbreaks in the last 3 years out of my own pockets because of lack of funding, not a great feeling being in the position of deciding whether to shut off an entire countries disease outbreak alert system or feed your kids that month… but that’s all gotten better in the last year as I went begging to a few doors outside the org I work for.

                                                                                                                                                                                Most of the time I just keep my head down in the back room of a little rented farmhouse in the countryside, support our countries 24/7 365 days a year, write code furiouslyday in day out and slink around in the shadows on lobste.rs.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  How much are those app servers running you?

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    During really busy months it can run just shy of 1000USD/mo. We’re hopefully going to get that down quite a bit in the next year as we’re releasing a desktop app that offloads some of the hard work to the users machine as well as deploying portions of the code written in rust compared to our currrent python heavy deployments.

                                                                                                                                                                                  2. 1

                                                                                                                                                                                    That’s incredible. You are a hero.

                                                                                                                                                                                    1. 1

                                                                                                                                                                                      Not at all. I just write code for the most part and answer a lot questions and write scripts to load/export data. The people that have to be out in the field, in the clinics, doing contact tracing for Ebola and stuff. Those people are just amazing. Stronger people do not exist considering the things they see and experience on a daily basis.

                                                                                                                                                                                    2. 1

                                                                                                                                                                                      Wow. Very impressive. Maybe you should post a link for donations.

                                                                                                                                                                                      1. 1

                                                                                                                                                                                        I don’t think I’m allowed to do stuff like that. Its sort of any funding I get has to probably be approved by the org. We’re hopefully going to open source the whole system sometime in the new year so closer to that I might write up an article to see if there’s any interest in contributing or helping with funding. If I took donations I’d have to pay tax and all that on them I think and that would probably be a nightmare come tax time… I also feel bad that people give money and a chunk of that is immediately lost to taxes…

                                                                                                                                                                                        1. 1

                                                                                                                                                                                          Probably a way to give to the UN.

                                                                                                                                                                                          1. 1

                                                                                                                                                                                            Yeah I’d need to do some digging on how it would work. You could give to the Org and stipulate that the funds have to go to the project but you’d have to be giving a not inconsiderable amount I think to have that stipulation and even then it would go to the department I think and probably not make it down to the actual project itself and as soon as it passed to me it would still taxed still as income.

                                                                                                                                                                                            The thing i would ultimately jump for joy for would just be 1-2 more devs more than just me and maybe an admin person. At least then I could go to bed at a reasonable hour, and I wouldn’t be so alone on the project all the time. I get paid well when I get paid so I can’t complain on that front but if I had a commitment of money for an extended period of time I could use for some additional staff I’d probably bawl my eyes out for a week straight out of sheer joy.

                                                                                                                                                                                1. 2

                                                                                                                                                                                  What’s wrong with current’s implementation written in ANSI C? Seems like a waste of human resources, in my opinion.

                                                                                                                                                                                  1. 32

                                                                                                                                                                                    What’s wrong with current’s implementation written in ANSI C?

                                                                                                                                                                                    If you want to script a Go application, it’s a hell of a lot easier if the VM is native instead of trying to fight with C interop issues.

                                                                                                                                                                                    Seems like a waste of human resources, in my opinion.

                                                                                                                                                                                    Incredible that with zero insight into the business requirements or technical motivations behind a project, you still consider yourself qualified to judge the staffing allocations behind it

                                                                                                                                                                                    1. 7

                                                                                                                                                                                      Incredible that with zero insight into the business requirements or technical motivations behind a project, you still consider yourself qualified to judge the staffing allocations behind it.

                                                                                                                                                                                      You are definitely right about this.

                                                                                                                                                                                      If you want to script a Go application, it’s a hell of a lot easier if the VM is native instead of trying to fight with C interop issues.

                                                                                                                                                                                      As @tedu commented, I understand the motivation behind it, thanks for pointing it out too.

                                                                                                                                                                                      1. 2

                                                                                                                                                                                        They give some further explanation of the technical requirements that led them not to pick up Shopify’s version here, which I found intersting: https://github.com/Azure/golua/issues/36.

                                                                                                                                                                                        The TL;DR is they’ve been scripting Helm for a while using another Lua-in-Go engine, but it isn’t updating to 5.3, which they very much want, along with closer conformance to the Lua spec, plus they have some design ideas that they feel would make debugging etc easier.

                                                                                                                                                                                        1. 3

                                                                                                                                                                                          To each their own, but I’m a little perplexed that strict 5.3 conformance would be a driver. I’m using gopher-lua which is like 5.1 but not exactly, but it’s just fine. What I need is some scripting language to embed, it’s close enough to lua it’s not really learning a whole new whatever, and it interfaces nicely with go (nicer than a strict port of the stack api would).

                                                                                                                                                                                          I don’t know jack about helm, but after seven seconds of research I can’t reverse engineer a requirement for lua 5.3 accept no substitutes.

                                                                                                                                                                                          1. 1

                                                                                                                                                                                            It’s possible that they have some need of true 64-bit integers, which only came to Lua in 5.3. That’s the only important change I can think of.

                                                                                                                                                                                    2. 5

                                                                                                                                                                                      Writing memory-safe C is an artform and a craft.

                                                                                                                                                                                      Writing safe code with Go is something a trained orangutang can do.

                                                                                                                                                                                      1. 3

                                                                                                                                                                                        Lua bytecode is memory-unsafe in the reference implementation, for starters.

                                                                                                                                                                                        1. 2

                                                                                                                                                                                          Do you have a link or more details?

                                                                                                                                                                                          1. 1

                                                                                                                                                                                            My upload server was down, sorry I didn’t link this originally. Here’s a slide deck (pdf) of a presentation I gave at Houston Area Hackers with almost all the details you could want.

                                                                                                                                                                                            I still have to update this; there are quite a few more tricks you can pull. I ended up creating a pure-Lua dlsym() implementation for the project that spurred all this research.

                                                                                                                                                                                            1. 2

                                                                                                                                                                                              Hmmm… Do reimplementations like this one remove support for lightuserdata then? I’m having a hard time imagining a lua interpreter for arbitrary bytecode that supported lightuserdata but could nevertheless guarantee that those pointers were safe.

                                                                                                                                                                                              1. 1

                                                                                                                                                                                                Yeah. In gopher-lua, userdata actually looks kind of like lightuserdata, which doesn’t exist. When both host and lua state share an allocator, the difference becomes less meaningful.

                                                                                                                                                                                                1. 1

                                                                                                                                                                                                  Looks like it’s unimplemented. lightuserdata is a pretty damn leaky Lua API. Bytecode itself is even more leaky in the Lua C implementation - arbitrary Lua bytecode is as expressive as C. No surprise that MS seemingly wanted to make an API/ABI compatible implementation that fixed some unnecessary leakiness.

                                                                                                                                                                                          2. 2

                                                                                                                                                                                            It’s annoying to interface with from go code.

                                                                                                                                                                                            1. 1

                                                                                                                                                                                              seems like something that should be fixed in Go rather than reimplementing everything written in C.

                                                                                                                                                                                              1. 1

                                                                                                                                                                                                For better or worse, it’s a result of deliberate design trade-offs in go, so it’s unlikely to change.

                                                                                                                                                                                          1. 9

                                                                                                                                                                                            I like how everyone dunks on C but like, we have four or five C alternatives right now that are viable, and numerous compilers and static analyzers and such that are fully capable of catching the bulk of problems that you have in C.

                                                                                                                                                                                            Not just that, but the problems they list are relatively simple to solve code-wise. Let’s take a look:

                                                                                                                                                                                            “use of uninitialized memory” -> just use calloc and memset on shit before you use it, and assign all of your variables default values. sparse or cppcheck are quite capable of catching these and will point them out for you.

                                                                                                                                                                                            “type confusion” -> Literally any sane static analyser will catch this (and I think GCC will too a lot of the time with -pedantic -Wall), but really you shouldn’t assign between variables of different types anyway, unless you’re using a bona-fide conversion function (like, lrint and friends – which will point out bugs for you). Personally speaking I take this further and use large integer sizes and the appropriate size_t and friends wherever possible, anything else is just premature optimization TBH. Besides, the entire point of typedef is to guard against this sort of thing, though. Don’t use void *, use typedef XYZ * typename, and you will very rarely have this bug,

                                                                                                                                                                                            “use after free” -> This goes under “use of uninitialized memory”. Anything that can be NULL/-1, check it. Set to NULL/-1 when you free. Clang’s scan-build catches a lot of these and sparse and cppcheck are capable of catching the rest.

                                                                                                                                                                                            Also, from what I’ve read of security literature, most of the vulnerabilities come from things like, not sanitizing your input, allowing larger inputs than you have space for, etc. Those are programming problems that you can have in any language, including Python. While C does give you room to fuck up, it also gives you the tools to NOT do that. Use them.

                                                                                                                                                                                            Javascript is quite literally a bigger danger with regards to proliferation, pure language idiocy, and fuckup potential, because you actually cannot avoid those parts that are broken (Some of which are generally considered to include, Unicode, and Arrays). People regard C as a loaded shotgun, and then go program in Javascript which has almost an equivalent number of flaws, and which is beyond broken.

                                                                                                                                                                                            Not just that, but C had and continues to serve a (somewhat debatable) purpose in the embedded world, in kernel development, in drivers, and some other places. Javascript was arguably superceded when Scheme was invented, 20 years before Javascript was born.

                                                                                                                                                                                            1. 8

                                                                                                                                                                                              Good points on mitigations being available. On last paragraph, Ada has been around a long time, too, with stuff written in it having less defects. Same with Modula-3 at one point. Newer stuff like Rust can do embedded. There was even a C/C++ implementation done in Scheme for its benefits. D and Nim are contenders with advantages, too.

                                                                                                                                                                                              C’s time has passed on technical grounds. Best replacement should still integrate well with its ecosystem, though, for practical reasons.

                                                                                                                                                                                              1. 3

                                                                                                                                                                                                On last paragraph, Ada has been around a long time, too, with stuff written in it having less defects. Same with Modula-3 at one point. […]

                                                                                                                                                                                                Oh, indeed! However the main benefit to C as it is, is the lack of linguistic complexity. It’s easy to pick the Right Way To Do Things, there’s very little room for debate, except perhaps architecturally – i.e. where it matters. But in addition to that, the best feature of that linguistic complexity is that a) it’s an easy language to remember, and b) it’s an easy language to hold in your head. It doesn’t require a ridiculously huge parser and it’s ‘easy’ to port (at least, it was, heh).

                                                                                                                                                                                                C’s time has passed on technical grounds.

                                                                                                                                                                                                I disagree :)

                                                                                                                                                                                                The main contender, Rust, not only has a ridiculously bloated stdlib (on part with Common Lisp’s with how lost you can get in it), and AFAIK still produces pretty large binaries. In addition it pushes a specific method of building on you, which really isn’t favourable to me.

                                                                                                                                                                                                Personally I’d like to see a systems-level language with the syntax of Lisp or Scheme and the philosophy of C, just with a more robust (but hackable) type system.

                                                                                                                                                                                                1. 3

                                                                                                                                                                                                  re linguistic complexity. There’s many of you that say that. I think you all pick a subset you need with coding style you can work with. That might keep it simple for you. The language itself isn’t simple as I said in the counter to vyodaiken. The people that were modeling and analyzing other languages took something like 40 years to do the same for subsets of C. Even experts like vyodaiken argue with other experts about the language details here on threads about pretty, basic stuff. That doesn’t seem simple.

                                                                                                                                                                                                  re Rust criticisms. Thanks for sharing them. I know Rust isn’t the end all. If anything, there’s still room for stuff that’s more C-like and flexible to help folks that don’t like Rust. I keep mentioning languages like Cyclone and Clay to give them ideas.

                                                                                                                                                                                                  re Lisp/Scheme. There’s two I know of in that direction: PreScheme and ZL. ZL had the most potential if combined with C tooling. Website here. If it doesn’t already, its own implementation probably could be updated to better tie-into popular forms of Scheme like Racket and HtDP.

                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                    re Lisp/Scheme. There’s two I know of in that direction: PreScheme and ZL

                                                                                                                                                                                                    I think bitc was quite promising too (from afar, I’ve never actually played with it). I don’t know what happened to it, its website seems down.

                                                                                                                                                                                                    1. 4

                                                                                                                                                                                                      There was Retrospective Thoughts on BitC in 2012. Mail archive is down too, but you can use Internet Archive.

                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                        Thanks for that! I’ll have to take out some time to read it, it’s quite long.

                                                                                                                                                                                                      2. 3

                                                                                                                                                                                                        He was on a row with EROS, COYOTOS, and BitC. Then, Microsoft poached him. (sighs) The best development in high-assurance on language side in recent times is COGENT. They did two filesystems in it. Although paper talks general purpose, O’ Connor showed up on Reddit and HN saying it’s only for stuff like filesystems. He wouldn’t answer follow-up questions about that. So, it’s interesting at unknown usefulness.

                                                                                                                                                                                                      3. 1

                                                                                                                                                                                                        The language itself isn’t simple as I said in the counter to vyodaiken. The people that were modeling and analyzing other languages took something like 40 years to do the same for subsets of C.

                                                                                                                                                                                                        As I said, syntactically. The entire C grammar can fit in three pages. The base C library is like 20 pages and fits into the end of K&R next to the grammar. If you want more functions there’s posix, 99% of which is part of the base operating system.

                                                                                                                                                                                                        You’re right that C-the-implementation isn’t simple. But at that level there are very few simple things, anyway. There are lots of approaches to choose from implementation-wise, for threads, etc. And not to mention the reality of the machine underneath, which doesn’t give a crap about what you think about it.

                                                                                                                                                                                                        With regards to program verification, you are indeed correct, but I’d argue the main problem with that was that C ended up being subjected to the mutagens of almost every single platform of the 1970s to 1990s, very few of which were standardised. The standards committee ended up having to backwards-support everything. That’s ignoring the fact that in certain cases they make it deliberately more difficult to standardize for the sake of improving optimization, or allowing optimizations that already exist in the wild.

                                                                                                                                                                                                        I was mulling it over after I wrote the above, and I think it’s useful to adopt a view of C as being forged by the pressures of being quick to write a compiler for, (and therefore relatively simple to understand how something was implemented (see: macro system, standard library – indeed, 99% of K&R is just teaching you C by reimplementing the standard library in tiny snippets of C)), and close enough to the machine that it’s easy to make optimization choices – you can generally (although it’s got harder with more advanced processors), just by looking at the C source, figure out the machine code produced. That’s where C’s power lies, and it’s something that other languages really do not know how to capture.

                                                                                                                                                                                                        Like, it’s one thing to be able to say, X is better than Y, but C adopts itself really well to showing you why, I guess. And I don’t think we can find a replacement for C until we figure out a language that captures both of those features.

                                                                                                                                                                                                      4. 1

                                                                                                                                                                                                        In addition it pushes a specific method of building on you, which really isn’t favourable to me.

                                                                                                                                                                                                        tbh this is my major objection to rust as well. For all C’s build “process” gets maligned, it is very easy to swap in different tools.

                                                                                                                                                                                                        1. 1

                                                                                                                                                                                                          However the main benefit to C as it is, is the lack of linguistic complexity

                                                                                                                                                                                                          Say what now?

                                                                                                                                                                                                          https://hackernoon.com/so-you-think-you-know-c-8d4e2cd6f6a6

                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                            I believe you are replying to the wrong person.

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              Oops.

                                                                                                                                                                                                        2. -1

                                                                                                                                                                                                          However the main benefit to C as it is, is the lack of linguistic complexity. It’s easy to pick the Right Way To Do Things, there’s very little room for debate, except perhaps architecturally – i.e. where it matters.

                                                                                                                                                                                                          Excellent point and exactly why the Crappy Pascal initiative also known as the ISO C Standard has been so detrimental to C.

                                                                                                                                                                                                          1. 4

                                                                                                                                                                                                            I don’t know if you’ve ever worked with pre-ANSI-C code, but given the choice between that and ANSI-C, ANSI-C wins if just for function prototypes.

                                                                                                                                                                                                            What I don’t like about the standard is the tortured language so that everything from signed-magnitude to 2’s-complement, 8-bit to 66-bit, can be supported. That may have been a valid compromise for C89, but is less and less so as time goes on [1]. The major problem with C now is the compiler writers trying to exploit undefined behavior to increase speed, to the point that formerly valid C code now breaks.

                                                                                                                                                                                                            [1] Byte addressable, 2’s-complement won. Get over it C Standards Committee!

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              The major problem with C now is the compiler writers trying to exploit undefined behavior to increase speed, to the point that formerly valid C code now breaks.

                                                                                                                                                                                                              I think the major problem is people compiling with -O3 and then complaining that compilers are trying to make their broken code fast.

                                                                                                                                                                                                              1. 0

                                                                                                                                                                                                                The standard is full of contradictions and obscurities. Compiler writers treating the standard as if it were some shrink wrap contract that they could exploit to evade every obligation to their users is simply wrong. The code you complain is “broken” is not even broken according to the standard and is common in things like K&R2. It’s ridiculous to claim that exploiting loopholes in murky standard written by a committee that seems to have no idea what they are doing is somehow justifiable.

                                                                                                                                                                                                              2. 1

                                                                                                                                                                                                                You may get your wish soon. From https://herbsutter.com/2018/11/13/trip-report-fall-iso-c-standards-meeting-san-diego/:

                                                                                                                                                                                                                … all known modern computers are two’s complement machines, and, no, we don’t really care about using C++ (or C) on the ones that aren’t. The C standard is likely to adopt the same change.

                                                                                                                                                                                                                1. 0

                                                                                                                                                                                                                  I think K&R2 is basically C at its best and that is ANSI. I even like restrict, although God knows the description of it in the standard reads like the authors were typing it on their phones while working at the Motor Vehicle Bureau. But there are programs in K&R2 that are not valid programs according to the current interpretation of the current incarnation of the standard.

                                                                                                                                                                                                          2. 3

                                                                                                                                                                                                            esides, the entire point of typedef is to guard against this sort of thing, though.

                                                                                                                                                                                                            The big flaw in typedef is that there is implicit type conversion between e.g. typedef int metric x and typedef int english y that permits x = y etc. There should be a flag in C to give warnings on all implicit type conversions. and a strong typedef (although the struct method works well too )

                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                              To solve this thing, just do that thing!

                                                                                                                                                                                                              If it’s so simple and yet so easy to forget… why don’t we just automate it? 😉

                                                                                                                                                                                                              1. 0

                                                                                                                                                                                                                why don’t we just automate it?

                                                                                                                                                                                                                If you read what I have written, that is what I said. scan-build+sparse+cppcheck+valgrind will catch 99% of the errors mentioned in the article, and they take only about 5 seconds to run.

                                                                                                                                                                                                                1. 7

                                                                                                                                                                                                                  Sounds to me like you’re choosing a language that requires boilerplate, then installing tools to scan for missing boilerplate.

                                                                                                                                                                                                                  For something as important as memory safety, it seems shortsighted to arrive at such a solution. But to each their own: if a certain workflow helps you produce safe code, then I won’t complain it puts the cart before the horse.

                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                    Valgrind requires you to exhaustively test your application. That’s not a five-second job.

                                                                                                                                                                                                                2. 1

                                                                                                                                                                                                                  Nulling things only works if you have a single pointer to the memory, and pass that pointer by reference when you need it

                                                                                                                                                                                                                1. 4

                                                                                                                                                                                                                  I think the bigger problem is that people extend the security expectations of tools and then get surprised that they weren’t designed with it in mind. ImageMagick is a great tool, but I assume it originally thought you’re manipulating your own images, not untrusted ones. People started expecting that and are surprised their needs are not magically met.

                                                                                                                                                                                                                  1. -1

                                                                                                                                                                                                                    Good point. Using security issues in ImageMagik as evidence of flaws in C is beyond stupid.

                                                                                                                                                                                                                  1. 5

                                                                                                                                                                                                                    So go implementation of Lua used to script Helm to control Kubernetes to distribute containers to virtual machines that will execute the applications inside the containers within jails. dizzying.

                                                                                                                                                                                                                    1. 2

                                                                                                                                                                                                                      I think the problem is something else entirely. The field of computer science has a problem with trying to define away hard problems. C/C++ are hugely succesful languages for good reasons. Making better languages or, more usefully, improving code analyzers for C, writing better C code, these things are hard. Declaring that C/C++ suck and that there is a magic cure is a lot easier.

                                                                                                                                                                                                                      1. 8

                                                                                                                                                                                                                        Although I agree with tool development and use being high priority, I think your last point about C/C++ suckage is off since we can prove it does for verifiable software. Just gotta look at effort vs what’s gained.

                                                                                                                                                                                                                        1. Pascal-, ML-, and LISP-like languages. Early work on formal semantics for verification was done on these with fairly quick results. They got some use out of them with 1970’s-1980’s era methods. It was also easiest to certify the code generators and such. SPARK generated the most ROI. The common denominator was they were designed in a way to make formalization easier and verification feasible.

                                                                                                                                                                                                                        2. BCPL was what random features compiled on a terrible computer. It was tweaked and extended in ad hoc way with strong limitations and dependencies imposed by a less terrible computer. It was extended with no verification in mind for years. The result was a language so hard to formalize that CompSci folks needed 40 years of advancements in provers and hardware to do simple, Pascal-like subsets. Another 10-20 years before realistic semantics appear… maybe 2 or 3 of them. Those are either subsets or leave off compiler extensions.

                                                                                                                                                                                                                        The comparison indicates that, compared to alternatives (esp Gypsy or SPARK), C was and still is ridiculously hard to formalize for safety analysis since it was (a) not designed for verification and (b) has significant, incidental complexity tied to its ad-hoc development on PDP-11. It does suck for anyone wanting to be certain code is ssfe with minimal work. Even more for folks building those tools.

                                                                                                                                                                                                                        1. 0

                                                                                                                                                                                                                          I don’t believe anyone knows how to do verification, so nobody knows how to make verification easier. I’m super skeptical about it being easier to build a mathematical model of ADA than KR-C (as opposed to the so-called C-standard C, otherwise known as Botched Pascal)

                                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                                            Nah, they built models of Pascal-like languages most of the time. One generated Ada. SPARK was a subset of Ada they could handle plus verification-related extensions.

                                                                                                                                                                                                                        2. 7

                                                                                                                                                                                                                          What’s wrong with putting the work in, making a better language (arguable), then declaring it a cure?

                                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                                            C/C++ are hugely succesful languages for good reasons

                                                                                                                                                                                                                            Yes.

                                                                                                                                                                                                                            C, because it was the systems language of Unix, and that ended up getting popular.

                                                                                                                                                                                                                            C++, because of its near perfect backwards compatibility with C while making it better.

                                                                                                                                                                                                                            writing better C code, these things are hard

                                                                                                                                                                                                                            So hard that basically nobody manages.

                                                                                                                                                                                                                            Declaring that C/C++ suck

                                                                                                                                                                                                                            Compared to alternatives.

                                                                                                                                                                                                                            there is a magic cure is a lot easier

                                                                                                                                                                                                                            I don’t think anyone is saying there’s a magic cure. Only that there are better ways of writing software than the current ones that would eliminate an entire class of bugs.