1. 14
  1.  

  2. 4

    An interesting aspect of safety, as defined by ISO standards, is that software cannot be safe or unsafe. It must be combined with hardware to become a “machine”. Machines can be safe or unsafe. Software in itself is incapable of hurting people.

    1. 1

      got any more general info on this short of a literal spec? like a blog article?

      1. 1

        I googled around but could not find anything worthwhile. Officially it starts at ISO 26262 for me but the definitions there are based on another standards and they are all not (completely) available online. They are not very readable either. I got my education from plenty of company-internal trainings.

    2. 2

      I think the article is missing another point regarding the lack of adversaries. Nobody blames the aircraft designer when an airliner is shot down with military weapons.

      On a separate note I find the fact that we don’t know how to build safe software even when we’re given the resources is shameful. The government could spend a lot of money on a safe digital voting platform, but we still wouldn’t trust the outcome.

      1. 4

        On a separate note I find the fact that we don’t know how to build safe software even when we’re given the resources is shameful. The government could spend a lot of money on a safe digital voting platform, but we still wouldn’t trust the outcome.

        That’s not true, though. Dijkstra’s method (1960’s onward), VDM (1970’s), Cleanroom (1980’s), B-Method (1980’s), and Correct-by-Construction (maybe 1990’s) helped teams deliver software with very, low, defect rates. Companies some even issued warranties for their code. They were usually on commercial schedules without time and staff to apply heaviest methods, like full proofs of correctness. So, they aren’t even the upper bound. They just applied known-good methods to design and analyze the software with bug counts orders of magnitude less than most software. CbC also used SPARK Ada to prove the most critical parts free of code-level errors. Rockwell-Collins and Galois are doing it for hardware/software combinations.

        So, we do know how to make low-defect software. Those with the time and money don’t care enough to spend it due to no liability. The FOSS people just don’t want to do it. Most don’t care about safety or security that much. The ones that claim to want safe and secure projects usually use the most unsafe methods claiming it’s a programmer’s responsibility to juggle it all in their head. This is unique mindset compared to most fields where they adopt every tool they can afford to boost productivity and lower risks. The biggest uptake of these methods are in regulated markets (i.e. safety-critical, smartcards) and a subset of CompSci that pushes state-of-the-art for safety and security. Even in safety-critical, the uptake is still minimal since they have a different mindset about how to do correctness which is [justifiably] slow to change.

        So, we know what works, most software avoids it due to no liability, and regulated markets do stuff like it due to regulations. That tells me we need regulations and/or court liability for any commercial software that has errors we can prevent using hardly any effort and/or automated means. That includes FOSS but only for commercial use. That will as a side effect force FOSS-using companies to invest in its quality. Like what happened in DO-178C market, you’ll see piles of pre-certified components and tools making safety easier pop up to reduce costs of projects. Also, it’s funny writers like OP talking about this never bring up success of DO-178C regulations. Probably didn’t know it existed.

        1. 3

          that we don’t know how to build safe software even when we’re given the resources is shameful

          I feel like that’s a bit like saying ‘that we don’t know how to factorise prime numbers quickly is shameful’. There’s no reason to assume that it is possible to have safe digital voting that preserves all the things we need in a voting system, namely that you can be sure your vote was counted, you can’t prove how you voted, you can be sure that nobody’s vote was counted twice, you can be sure that nobody voted twice, you can be sure nobody else knows what you voted for, etc.

          The government could spend a lot of money on a safe digital voting platform, but we still wouldn’t trust the outcome.

          Essentially, no, they couldn’t. The problem of voting systems can’t just be solved by throwing money at them. Digital voting is fundamentally not something that will ever be possible, for the simple reason that someone can stand over you while you vote, defeating the whole purpose of secret voting.

          1. 3

            Now I see how I’ve driven my argument through a cliff by touching on digital voting, since the subject has theoretical problems. What I meant was, if it were something that had a solution, we wouldn’t trust the thing those people who’ve won the contract build. The argument is, if the government decided to spare the resources and build a 250-story-high mega sky scraper, I’d expect them to find the right people to build it. The thing probably wouldn’t collapse after all. But I’d never expect them to build something of that quality, if the mega artifact in question were software. We simply don’t have the established practices for building provably (and independently checkably) safe software, in any case even if there were people who could build that artifact, I’m pretty sure those ones wouldn’t get the contract, because we don’t have the means to identify those people either.

            1. 2

              But they can build highly reliable software at NASA/JPL etc. I think some of this may be the reliance on in-house or near-in-house engineering rather than contractors. The Fed contracting system weakness is particularly bad for software, although they have spectacular failures in other areas too.

            2. 2

              Of course voting by paper ballot has the same problem. You have just proved that voting cannot be possible.

              1. 2

                Paper ballots definitely don’t have the same problems. With paper ballots, you are sure your vote was counted, you can’t prove how you voted, you are sure that nobody vote was counted twice, you are sure nobody voted twice, and you are sure nobody can see what you voted for. The system is designed to sort these issues out, quite well.

                1. 2

                  Paper ballot: physical presence at the scene of the crime for a few thousand votes.

                  Electronic voting: might be able to get with it, one person can sway an entire country. You only need access to a few machines at any point before the event.

                  The risk/reward for an attacker heavily favors paper ballots. Nothing is absolutely safe, there are people who disrupted paper voting too. They got a lot less for their actions and the rest of the system was safe.

                  1. 1

                    the problem with paper voting is that it is now usually tabulated by machine. But your points are well taken. Needs a better design than anyone has demonstrated yet to get the same level of security with purely electronic systems. There are tradeoffs, however. For example. Travis Country TX uses electronic voting to enable any county voter to vote at any polling place - the electronic system is able to produce the many different ballot configurations on demand which vary by location. This is especially useful because we have one of the most gerrymandered maps in the world imposed on us.

                2. 2

                  for the simple reason that someone can stand over you while you vote, defeating the whole purpose of secret voting.

                  You could have a duress code or something for that. The reason it’s better to use paper is there’s simply fewer attacks that are possible and more people are mentally capable of auditing it. Less risk, more democratic.

                  1. 1

                    A duress code that they could see you putting in.

                    There are fewer attacks possible, for example you can actually have secret voting.

                    1. 1

                      Duress codes would be unique to the voter with the system showing the vote was tallied. Obviously. Otherwise, it wouldnt work.

                      1. 1

                        How would you guarantee the secret delivery of the duress code? Whatever you use for duress code delivery, why not use that for vote delivery? And repeat.

                        1. 1

                          Off top of head, they get it during registration. They dont bring it with them: they memorize it voting day and destroy it. If they cant, they can bring it with them labelled in an obfuscated way they understand but not the crooks. Some people might for whatever reason have the code on them. That case will range from ineffective (attackers see code) to effective (most not carrying it deterred manipulation).

                  2. 2

                    Digital voting is fundamentally not something that will ever be possible

                    I understand what you’re getting at, but keep in mind that I read this sentence while also doing my due diligence research on who I’ll be voting for in the upcoming Estonian parliamentary election from my home in Australia. I don’t mean to suggest that it’s unflawed or perfect, but digital voting is not only “not possible”, but has been implemented and in use for more than a decade in some countries.

                    1. 1

                      The article you link to has a large criticism section, and nothing in it seems to explain how they’ve got past any of the issues of online voting.

                      Firstly, there’s absolutely no way that your vote can be anonymous if it’s clearly recorded, which it has to be in order to be able to be erased by the same person making a subsequent vote.

                      Secondly, they haven’t solved the problem that such a system destroys the most important attribute in a voting system, above all other attributes, which is to have a secret ballot. If someone can watch you vote, then they can force you to vote a certain way. They can blackmail you, they can bribe you, etc. That’s antithetical to a fair voting system.

                      1. 1

                        I’m responding only to the claim that digital voting is impossible by pointing out that it is happening now and has been happening for decades. Thanks!

                        1. 1

                          I said that secure digital voting is impossible.

                          1. 2

                            I said that secure digital voting is impossible.

                            Digital voting is fundamentally not something that will ever be possible

                            ¯\_(ツ)_/¯

                            While I’m here, though: no, it’s not provably anonymous. We do need to trust the procedure that after vote deduplication, the remaining encrypted votes are passed onto the systems with the actual keys for decryption without any identifying data. This is why I said it’s not unflawed or perfect. At the end of the day, we do have to put some trust in the government running the show, rather than in cryptography. Not ideal, but it still does effect digital voting. You can’t just stick your head in the sand and shout that digital voting is impossible while it’s happening, even if it’s flawed.

                            But at the same time, due to this property, your other point is moot. The ability to recast your vote means blackmail, bribes, etc. cannot be committed effectively.

                            1. 1

                              I thought it was pretty obvious that digital voting that isn’t secure was not really digital voting.

                              But at the same time, due to this property, your other point is moot. The ability to recast your vote means blackmail, bribes, etc. cannot be committed effectively.

                              Wrong. Someone can stand over you until the voting period is over. e.g. an abused spouse.

                              1. 2

                                It’s hard to coerce a vote at scale - i.e. enough to swing an election.

                                Individual abuses of voter’s rights are to be deplored, but in the grand scale of things (electoral integrity) don’t really register.

                                1. 2

                                  It’s hard to coerce a vote at scale. But even given theoretical flaws in a paper voting system, exploiting those flaws usually requires thousands of people to collaborate, and even one breaking their silence means the whole scheme is undone.

                                  But if there’s a security issue with digital voting, then the entire system is probably COMPLETELY broken and cannot be trusted at all, and it can be all done from another country by a foreign government, in secret, or even by a single person.

                                  1. 1

                                    It does with smaller elections, esp with low participation. People have lost city and county elections by a tiny number of votes. In once case, a politician was said to have lost just by not voting for himself because he was too busy getting others’ votes. I didn’t verify that but would be hilarious.

                                  2. 1

                                    I thought it was pretty obvious that digital voting that isn’t secure was not really digital voting.

                                    This leads to the claim that 31.5% of voters in the 2017 Estonian municipal elections gave their vote online, but that they did not vote digitally.

                                    Wrong. Someone can stand over you until the voting period is over. e.g. an abused spouse.

                                    They would need to stand over you at all times, including preventing you from attending a physical polling booth location. This is probably an unavoidable consequence.

                    2. 1

                      I’d like to hear more about the history of aviation safety. I can’t imagine that airplanes were always so safe. If I’m right about that, then the high ratio of effort to accidents (3 safety board member years/per incident) might be at least partially the product of aviation safety, not just the cause.

                      1. 1

                        By contrast, the aircraft which became the Boeing 787 Dreamliner was announced in 2003, based on technology which had been in development since the late 1990s. The first production aircraft was delivered eight years later, in 2011. And the planes have an expected operational lifetime of something like 40 years.

                        Dreamlner has a history of software problems.

                        One possibility is that software is just a young engineering field, suffering problems like those in early jets before they figured out about metal fatigue. A contributing issue is that software as a field is comprehensively infected by “premature formalization”. Once best practices have been established through trial and error and research, it’s good to formalize process, but safety critical software process is formalized by guidelines that are just made up

                        1. 3

                          One possibility is that software is just a young engineering field, suffering problems like those in early jets before they figured out about metal fatigue.

                          Nah, they ignore it. Easy example was all kinds of problems that came with C due to its unsafety and being designed for limitations of a PDP-11. So, Ada’s inventor systematically looks for all the error classes, designs a language-level mitigation for as many as possible, and delivers a solution. Most don’t want to use it. Same with Modula-3 which was simpler. For Ada, the problems it had have mostly been fixed. Most still don’t want to use it. Hell, many didn’t even know such a language existed before Rust made similar claims. They know about Rust which mitigates common heisenbugs, too. Most low-level programmers still aren’t using it.

                          The aviation comparison would be more in line with software development if some folks insisted on using stuff that could fatigue saying you just gotta review it hard, put in extra checks or mitigations, and so on. There’s nothing wrong with the metal, though. Let’s not use anything that automatically prevents this problem with a small installation and performance penalty. The aviation designers would be like, “What the hell are you people thinking?!”

                        2. 1

                          Factors:

                          1. In terms of engineering, “software” is not a thing comparable to “planes” or “elevators”. Software “can be anything” and so a random thing “done with software” is a random thing that might have no particular engineering practices involved in it’s construction. Planes now use software for example but the pre-software engineering practices in place seem to have kept things as safe as previously.

                          2. Software in the generic does not necessarily follow the logic of a thing bounded by time and space. So our natural human intuitions about vulnerabilities don’t necessarily apply. Finding and plugging a vulnerability is like solving a complicated puzzle or math theorem. No one really can predict when that will occur.

                          3. The intuition of managers and decision makes about software is even worse than the intuition of actual software engineers. All the real engineering fields that exist don’t rely on just the hope that upper managements will make allowances for the necessary means to create reliable software - they include field specific regulation. But back to the point of “software” not being one thing (relative to engineering) but a million things. You can apply software to areas where bugs and failure have little impact, you can apply software to areas where it has immediate impact and you apply it to areas where the impact is felt to by others or only appears over time. The people controlling the purse things wind-up concerned, at best, with immediate impact situations. Considers engineers certainly “reinvent the wheel” on a regular basis - engineer need to engineer the wheels used on a given plane for the particular tight constraints involved in flight. Sure, you could stick heavy truck tires on a 747 and it might land safely. The problem is a purse-string pullers may well be happy with the trade-offs of a thing that mostly works for a cheaper price. If people could build skyscrapers with no immediate consequences when they collapsed, would they do so? Of course they would - we occasionally things like that in less regulated areas (China in the 2000s, say).