1. 8

    yet in many respects, it is the most modern database management system there is

    It’s not though. No disrespect to PostgreSQL, but it just isn’t. In the world of free and open source databases it’s quite advanced, but commercial databases blow it out of the water.

    PostgreSQL shines by providing high quality implementations of relatively modest features, not highly advanced state of the art database tech. And it really does have loads of useful features, the author has only touched on a small fraction of them. Almost all those features exist in some other system. But not necessarily one single neatly integrated system.

    PostgreSQL isn’t great because it’s the most advanced database, it’s great because if you don’t need anything state of the art or extremely specialized, you can just use PostgreSQL for everything and it’ll do a solid job.

    1. 13

      but commercial databases blow it out of the water

      Can you provide some specific examples?

      1. 16

        Oracle has RAC, which is a basic install step for any Oracle DBA. Most Postgres users can’t implement something similar, and those that can appreciate it’s a significant undertaking that will lock you into a specific workflow so get it right.

        Oracle and MS-SQL also have clustered indexes. Not what Postgres has, but where updates are clustered as well. Getting Pg to perform sensibly in this situation is so painful, it’s worth spending a few grand to simply not worry about it.

        Ever run Postgres on a machine with over 100 cores? It’s not much faster than 2 cores without a lot of planning and partitioning, and even then, it’s got nothing on Oracle and MS-SQL: Open checkbook and it’s faster might sound like a lose, but programmers and sysadmins cost money too! Having them research how to get your “free” database to perform like a proper database isn’t cost effective for a lot of people.

        How about big tables. Try to update just one column, and Postgres still copies the whole row. Madness. This turns something that’s got to be a 100GB of IO into 10s of TBs of IO. Restructuring this into separate partitions would’ve been the smart thing to do if you’d remembered to do it a few months ago, but this is a surprise coming from commercial databases which haven’t had this problem for twenty years. Seriously! And don’t even try to VACUUM anything.

        MS-SQL also has some really great tools. Visual Studio actually understands the database, and its role in development and release. You can point it at two tables and it can build ALTER statements for you and help script up migrations that you can package up. Your autocomplete can recognise what version you’re pointing at. And so on.

        …and so on, and so on…

        1. 3

          Thanks for the detailed response. Not everyone has money to throw at a “real” enterprise DB solution, but (having never worked with Oracle and having only administered small MSSQL setups) I did wonder what some of the specific benefits that make a DBA’s life easier were.

          Of course, lots of the open source tools used for web development and such these days seem to prefer Postgres (and sometimes MySQL), and developers like Postgres’ APIs. With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.

          1. 7

            Not everyone has money to throw at a “real” enterprise DB solution

            I work for a commercial database company, so I expect I see a lot more company-databases than you and most other crustaceans: Most companies have a strong preference to rely on an expert who will give them a fixed cost (even if it’s “money”) to implement their database, instead of trying to hire and build a team to do it open-source. Because it’s cheaper. Usually a lot cheaper.

            Part of the reason why: An expert can give them an SLA and has PI insurance, and the solution generally includes all costs. Building a engineering+sysadmin team is a big unknown for every company, and they usually need some kind of business analyst too (often a contractor anyway; more £££) to get the right schemas figured out.

            Professional opinion: Business logic may actually be some of the least logical stuff in the world.

            lots of the open source tools used for web development and such these days seem to prefer Postgres

            This is true, and if you’re building an application, I’d say Postgres wins big. Optimising queries for dbmail’s postgres queries was hands down much easier than any other database (including commercial ones!).

            But databases are used for a lot more than just applications, and companies who use databases don’t always (or even often) build all (or even much) of the software that interacts with the database. This should not be surprising.

            With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.

            I’m not sure I disagree, but I don’t think this is a good thing. EnterpriseDB isn’t Postgres. Neither is redshift. Queries that work fine in a local Pg installation run like shit in redshift, and queries that are built for EnterpriseDB won’t work at all if you ever try and leave. These kinds of “hybrid open source” offerings are an anathema, often sold below a sustainable price (and much less than what a proper expert would charge), leaving uncertainty in the SLA, and with none of the benefits of owning your own stack that doing it on plain postgres would give you. I just don’t see the point.

            1. 3

              Professional opinion: Business logic may actually be some of the least logical stuff in the world.

              No kidding. Nice summary also.

              1. 0

                Queries that work fine in a local Pg installation run like shit in redshift

                Not necessarily true, when building your redshift schema you optimize for certain queries (like your old pg queries).

            2. 4

              And yet the cost of putting your data into a proprietary database format is enough to make people find other solutions when limitations are reached.

              Don’t forget great database conversion stories like WI Circuit Courts system or Yandex where the conversion to Postgres from proprietary databases saved millions of dollars and improved performance…

              1. 2

                Links to those stories?

                1. 1

                  That Yandex can implement clickhouse doesn’t mean everyone else can (or should). How many $100k developers do they employ to save a few $10k database cores?

                  1. 2

                    ClickHouse has nothing to do with Postgres, it’s a custom column oriented database for analytics. Yandex Mail actually migrated to Postgres. Just Postgres.

                2. 2

                  You’re right about RAC but over last couple of major releases Postgres has gotten alot better about using multiple cores and modifying big tables. Maybe not at the Oracle level yet bit its catching up quickly in my opinion.

                  1. 3

                    Not Oracle-related, but a friend of mine tried to replace a disk-based kdb+ with Postgres, and it was something like 1000x slower. This isn’t even a RAC situation, this is one kdb+ core, versus a 32-core server with Postgresql on it (no failover even!).

                    Postgres is getting better. It may even be closing the gap. But gosh, what a gap…

                    1. 1

                      Not to be that guy, but when tossing around claims of 1000x, please back that up with actual data/blogpost or something..

                      1. 6

                        You remember Mark’s benchmarks.

                        kdb doing 0.051sec what postgres was taking 152sec to complete.

                        1000x is nothing.

                        Nobody should be surprised by that. It just means you’re asking the computer to do the wrong thing.

                        Btw, starting a sentence with “not to be that guy” means you’re that guy. There’s a completely normal way to express curiosity in what my friend was doing (he’s also on lobsters), or to start a conversation about why it was so much easier to get right in kdb+. Both could be interesting, but I don’t owe you anything, and you owe me an apology.

                        1. 2

                          Thanks for sharing the source, that helps in understanding.

                          That’s a benchmark comparing a server grade setup vs essentially laptop grade hardware (quad-core i5), running the default configuration right out of the sample file from the Git repo, with a query that reads a single small column out of a very wide dataset without using an index. I don’t doubt these numbers, but they aren’t terribly exciting/relevant to compare.

                          Also, there was no disrespect intended, not being a native english speaker I may have come off clumsy though.

                          1. 1

                            kdb doing 0.051sec what postgres was taking 152sec to complete.

                            That benchmarks summary points to https://tech.marksblogg.com/billion-nyc-taxi-rides-postgresql.html which was testing first a pre-9.6 master and then a PG 9.5 with cstore_fdw. Seems to me that neither was fair and I’d like to do it myself, but I don’t have the resources.

                            1.  

                              If you think a substantially different disk layout of Pg, and/or substantially different queries would be more appropriate, I think I’d find that interesting.

                              I wouldn’t like to see a tuning exercise including a post-query exercise looking for the best indexes to install for these queries though: The real world rarely has an opportunity to do that outside of applications (i.e. Enterprise).

                        2. 1

                          Isn’t kdb+ really good at stuff that postgres (and other RDBMS) is bad at? So not that surprising.

                          1. 1

                            Sort of? Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C): Got some tall skinny table? Try mmaping as much as possible. That’s basically what kdb does.

                            What was surprising was just how difficult it was to get that in Pg. I think we expected, with more cores and more disks it’d be fast enough? But this was pretty demoralising! I think the fantasy was that by switching the application to Postgres it’d be possible to get access to the Pg tooling (which is much bigger than kdb!), and we massively underestimated how expensive Pg is/can be.

                            1. 3

                              Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C)

                              Well, kdb+ is columnar, which is pretty different than how most people approach naive database implementation. That makes it very good for some things, but really rough for others. Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).

                              Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.

                              But, yes, if your primary use cases favor a columnar approach, kdb+ will outperform vanilla postgres (as will monetdb, clickhouse, and wrappers around parquet files).

                              You can get the best of both worlds You can get decent chunks of both worlds by using either the cstore_fdw or imcs extensions to postgres.

                              1.  

                                which is pretty different than how most people approach naive database implementation.

                                I blame foolish CS professors emphasising linked lists and binary trees.

                                If you simply count cycles, it’s exactly how you should approach database implementation.

                                Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).

                                So I haven’t done that kind of UPDATE in any production work, but I also don’t need it: Every customer always wants an audit trail which means my database builds are INSERT+some materialised view, so that’s exactly what kdb+ does. If you can build the view fast enough, you don’t need UPDATE.

                                Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.

                                If I have data that I need horizontal grabs from, I arrange it that way in memory. I don’t make my life harder by putting it on the disk in the wrong shape, and if I do run into an application like that, I don’t think gosh using postgres would really speed this part up.

                    2. 3

                      Spanner provides globally consistent transactions even across multiple data centers.

                      Disclosure: I work for Google. I am speaking only for myself in this matter and my views do not represent the views of Google. I have tried my best to make this description factually accurate. It’s a short description because doing that is hard. The disclosure is long because disclaimers are easier to write than useful information is. ;)

                      1. 2

                        @geocar covered most of what I wanted to say. I also have worked for a commercial database company, and same as @geocar I expect I have seen a lot more database use cases deployed at various companies.

                        The opinions stated here are my own, not those of my former or current company.

                        To put it bluntly, if you’re building a Rails app, PostgreSQL is a solid choice. But if you’ve just bought a petabyte of PCIe SSDs for your 2000 core rack of servers, you might want to buy a commercial database that’s a bit more heavy duty.

                        I worked at MemSQL, and nearly every deployment I worked with would have murdered PostgreSQL on performance requirements alone. Compared to PostgreSQL, MemSQL has more advanced query planning, query execution, replication, data storage, and so on and so forth. It has state of the art features like Pipelines. It has crucial-at-scale features like Workload Profiling. MemSQL’s competitors obviously have their own distinguishing features and qualities that make them worth money. @geocar mentioned some.

                        PostgreSQL works great at smaller scale. It has loads useful features for small scale application development. The original post talks about how Arcentry uses NOTIFY to great effect, facilitating their realtime collaboration functionality. This already tells us something about their scale: PostgreSQL uses a fairly heavyweight process-per-connection model, meaning they can’t have a huge number of concurrent connections participating in this notification layer. We can conclude Arcentry deployments using this strategy probably don’t have a massive number of concurrent users. Thus they probably don’t need a state of the art commercial database.

                        There are great counterexamples where specific applications need to scale in a very particular way, and some clever engineers made a free database work for them. One of my favorites is Expensify running 4 million queries per second on SQLite. SQLite can only perform nested loop joins using 1 index per table, making it a non-starter for applications that require any kind of sophisticated queries. But if you think about Expensify, its workload is mostly point look ups and simple joins on single indexes. Perfect for SQLite!

                        1. 1

                          But MemSQL is a distributed in-memory database? Aren’t you comparing apples and oranges?

                          I also highly recommend reading the post about Expensify usage of SQLite: it’s a great example of thinking out of the box.

                          1. 1

                            No. The author’s claims “Postgres might just be the most advanced database yet.” MemSQL is a database. If you think they’re apples and oranges different, might that be because MemSQL is substantially more advanced? And I used MemSQL as one example of a commercial database. For a more apples-to-apples comparison, I also think MSSQL more advanced than PostgreSQL, which geocar covered.

                            And MemSQL’s in-memory rowstore serves the same purpose as PostgreSQL’s native storage format. It stores rows. It’s persistent. It’s transactional. It’s indexed. It does all the same things PostgreSQL does.

                            And MemSQL isn’t only in-memory, it also has an advanced on-disk column store.

                    1. 0

                      Calling the Fediverse “Mastodon” is like calling all email “Gmail” or worse, “Postfix”.

                      1. -4

                        The upgrade from TCP to QUIC

                        For a guy that says he knows protocols he certainly doesn’t know the OSI layers

                        1. 10

                          Are you sure about this? He specifically talks about moving off TCP to a layer 4+5 solution, UDP headers with QUIC inside.

                          1. 0

                            I’m very sure. He keeps conflating TCP with QUIC which are not at the same layers

                            1. 13

                              But they are. Ask yourself, what does a connection mean in networking context? Previously it was almost always a tcp connection as that’s what tcp does. Now it can be a non-tcp QUIC connection that does it’s own connection handling logic, multiplexing, in-order delivery, etc. That’s the whole point of QUIC-as-the-transport-layer thing at all.

                              People suggested to split QUIC-the-transport layer from HTTP/2 and this is essentially what happened. It’s a transport layer level thing with built-in TLS that can handle arbitrary application protocols on top of it, not just HTTP.

                              1. 5

                                They are at the same layer. I suppose one could imagine a QUIC connection as having two transport protocols (UDP and QUIC) but I just think of it as one most of the time. The reason UDP is there is just because it wouldn’t work over the internet any other way, but you could run QUIC on top of IP if you wanted.

                                1. 1

                                  You certainly could, but it would never work on the real internet because of middleboxes that will only pass TCP and UDP. This is also what is stifling SCTP adoption.

                                  The transport protocol is UDP not QUIC, so it would be good to end the ambiguity when discussing QUIC.

                                  1. 2

                                    There’s no reason why it couldn’t work one day even though it doesn’t work now. QUIC is a transport protocol. It provides all the features of a transport protocol. What do you call SCTP-over-UDP then? Just UDP?

                                    1. 1

                                      SCTP isn’t over UDP. I’m not aware of any implementation in the wild that attempts this. SCTP has its own implementation in OS kernels (Linux, FreeBSD) beside TCP and UDP. It’s not “over UDP”. But middlebox firewalls / shaping devices tend to drop any traffic that is not ICMP, TCP, UDP, or IPSEC which is why SCTP has never gained traction even though it is a superior protocol for many situations especially mobiles where seamless connection roaming between cellular and WiFi would be very much welcomed. Instead we have to live with “some services on iOS devices, for example, use MPTCP which is only supported by Apple services like Siri because very few servers on the internet have MPTCP support in their kernels”.

                                      edit: I’m not an expert on SCTP, but I’ve certainly never heard of it being used over UDP. Would be curious to learn more if you’ve got a source.

                                      edit2: correct acronym for Multipath TCP is MPTCP

                                      1. 3

                                        RFC 6951.

                                        1. 0

                                          Interesting. Is anyone actually using this in the wild or is it just a dead RFC?

                                          1. 3

                                            It’s implemented by the FreeBSD SCTP stack.

                                            1. 0

                                              Yeah, but is anyone actually using it? :) I know dteske was disappointed at all of the missing/broken dtrace hooks for SCTP in FreeBSD

                                              1. 8

                                                I don’t think that was the original argument. You claimed QUIC is not a transport protocol because it sits on top of UDP, but that’s just a consequence of how the internet works. I showed you how SCTP tried to work around the problems around NATs by doing exactly the same: transmitting packets over UDP.

                                                1. 4

                                                  Yes. WebRTC uses SCTP over UDP for its data streams. Google Hangouts, Facebook chat, and Discord all use WebRTC. So a non-trivial portion of internet traffic actually uses it. Further, in this usage it’s implemented with a user-mode library, just like QUIC currently is.

                                                  1. 1

                                                    Excellent, thanks for this info!

                                          2. 1

                                            And had Google said “your web site ranking will drop if we can’t reach your site via SCTP” then you can bet all those middle boxes would be patched, updated or replaced immediately!

                                            1. 4

                                              That only fixes web sites that care about their Google ranking. It doesn’t fix the middle boxes that sit in front of web browsers on corporate intranets and public wifi hotspots, because there’s no website to penalize. It also doesn’t do anything about the deep web sites that aren’t crawled by Google anyway, because you have to log into them.

                                              I strongly suspect that most of the middle boxes in question are being deployed on those things.

                                              1. 4

                                                Those middle boxes are affecting the clients not the servers. Nobody’s going to upgrade their corporate SSL proxy for QUIC if the fallback to HTTP/1.1 is still working fine.

                                1. 4

                                  they told me that slow VirtIO drivers for FreeBSD are a known issue. Not a big deal then, KVM was developed on Linux and sure Linux guest drivers are more optimized.

                                  bhyve also provides virtio devices, so this is important even without any Linuxes involved.

                                  11.2-RELEASE-p4

                                  I’d like to see 12.0-BETA4 in the comparison.. though most performance improvements might’ve been merged into 11..

                                  as written in the FreeBSD network optimization guide

                                  Not the guide you should be looking at. calomel.org > wiki.

                                  1. 1

                                    I think the fast-forwarding changes (if that is what you were thinking of) got merged into 11.x already.

                                    From https://wiki.freebsd.org/NetworkPerformanceTuning :

                                    Since FreeBSD 11.0, fastfordwarding was improved, renamed tryforward (no more break IPSec) and it’s the default method.

                                    1. 1

                                      No, stop. NEVER give calomel.org any traffic for any reason whatsoever. Do not send people there. Do not recommend anything on that site. Stop. Please.

                                      1. 1

                                        Uh, okay?? Would be nice to hear actual reasons though, not just “no stop never”. The guide was pretty helpful for me.

                                        1. 2

                                          Articles on the bsd router project are generally pretty high quality and usually have a description of what the tunable does.

                                          1. 1

                                            I will have a look

                                          2. 1

                                            If you even mention Calomel on the OpenBSD mailing lists, for example, the result is usually blacklisting or banning. The information on that site is usually quite inaccurate and/or out of date. Bad things happen when people follow guides on there especially as I’ve seen tuning suggestions listed that the author has no idea what they do.

                                            Just avoid it. Please. If you want accurate information come to the mailing lists after our documentation has failed to provide you with the info you need.

                                      1. 2

                                        We are looking for remote developers (worldwide):

                                        C/C++ developers to scale a proprietary chat server (IRC-like in some ways) Web frontend and backend (PHP, Rails, Javascript) Mobile devs (iOS and Android) Elixir developers to work on a new project and possibly migrate Rails stuff to in the future

                                        Anyone who knows a lot about audio/video streaming on the internet as well.

                                        I’m under a strict NDA and cannot reveal my employer unless we choose to interview you. Compensation is at the top of the industry, 100% guaranteed.

                                        1. 3

                                          Can you tell your company that strict NDAs like this are hurting them? Point to this post if you need to.

                                          Like, I have experience with all but the mobile stuff you mentioned, but I have no idea if I’d want to apply since your company won’t let you talk about it.

                                          1. 1

                                            How do you know it’s hurting them? They’re already reaching Lobsters-grade talent. Also, they could be in stealth mode and/or stepping on a patent minefield of some sort where secrecy helps them avoid losses that could either bankrupt them or keep them forever niche as a big player quickly gets parity. Who knows.

                                            The last sentence is the lure that will make it worth looking into for talented individuals who can live with the NDA.

                                            1. 3

                                              No, we as workers gain nothing from NDAs. They do not benefit us. They do not help us. We should fight them and speak out every time there is an opportunity to do so.

                                              1. 4

                                                I’m sorry, but I can only say so much for a very good reason. I should have given more details, though: our sites are in the Alexa Top 1000, we aren’t VC funded or a startup, and money is not a barrier to anything for us. Our challenges are extremely interesting in a part of the industry most people don’t get to see behind the curtain of. We operate at a scale most people would only dream of.

                                                It’s entirely possible you wouldn’t be under an NDA, but I am, and I have to honor that. And I will gladly honor that because this opportunity is better than anything I thought I could get in my lifetime.

                                                We always find the right people. That’s not a worry for us. If you want to actually learn more, message me and send me a resume.

                                                1. 1

                                                  Is it adtech?

                                                  1. 2

                                                    No, not adtech.

                                                2. 1

                                                  I agree they’re a bit worse for us and we should oppose them. Saying workers gain nothing goes too far: many gain a paycheck and interesting work they choose over non-NDA jobs. Either they’re dumb or found the employment gainful for their needs. I’m thinking the latter.

                                          1. 29

                                            I have friends who work for Red Hat who are Not Happy about this.

                                            My speculation is that clients of Red Hat will see at most slow change. IBM’s not going to toss the cash cow RHEL, and the various cloud software offerings are what they apparently bought it for. However, internally I think we’ll see a massive diaspora of talent as Red Hat becomes IBM-ified. (All claims to the contrary from either company’s PR are of course to be ignored completely. They have to say that, to stave off the employee flight as long as possible.)

                                            Hot take: I wonder what this will mean for SystemD? ;)

                                            1. 8

                                              I’m unfamiliar with IBM’s Linux strategy; why would this mean anything wrt systemd specifically?

                                              1. 5

                                                Nothing, it’s just a play on the (IMHO very wrong) meme that systemd is only as successful as it is because it had RedHat backing.

                                                IBM probably doesn’t even know what systemd is on the “we’re buying a huge company for 20 billion” plane.

                                              2. 6

                                                Employees are rarely excited about being acquired, and let’s face it, history has shown that’s it’s been bad for both customers and employees unless the company being acquired is going out of business.

                                                1. 12

                                                  Hot take: I wonder what this will mean for SystemD?

                                                  Can it be a hot take if it’s not even a take? This is inquisitive (not argumentative), which is good for discussion but probably bad if your goal was to have an opinion.

                                                  1. 3

                                                    hot question

                                                  2. 5

                                                    I’m out of the loop. Could you explain the systemd comment?

                                                    1. 14

                                                      systemd was originally written by Lennart Poettering and Kay Sievers who work at Red Hat.

                                                      1. 3

                                                        Is it still maintained by them as part of their jobs at Red Hat?

                                                        1. 5

                                                          Yes

                                                          1. 3

                                                            Lennart Poettering on Twitter this morning (:

                                                            As you all know we never have been fans of portability. It will come at no surprise that in light of the recent developments we will discontinue all non-S/390 ports of systemd very soon now. Please make sure to upgrade to an S/390 system soon. Thank you for understanding.

                                                            1. 1

                                                              Even POWER? ;)

                                                    2. 3

                                                      Hot take: I wonder what this will mean for SystemD? ;)

                                                      I’m pretty sure Facebook will keep developing it if nobody else does:

                                                      https://media.ccc.de/v/ASG2018-192-state_of_systemd_facebook

                                                      (disclaimer: I work there, though not on the team that works most with systemd – and this is of course my personal opinion)

                                                    1. 4

                                                      Go to System Preferences > Network > Advanced > DNS, add two entries to DNS Servers for 1.1.1.1 and 1.0.0.1 and remove any other server

                                                      Try doing this on any network that I maintain and you’ll find your DNS queries are being dropped. Allowing outbound traffic to any DNS server is not recommended. Well, allowing unrestricted outbound traffic is not recommended. It’s 2018. Don’t trust anyone or any device. Only allow out the traffic you need out.

                                                      1. 2

                                                        Honestly I think it’s bad advice just to tell people to “hey use this DNS server instead” anyway. It actually doesn’t protect your privacy by doing so, because anyone with tcpdump on a host between you and that DNS server can still record what you are looking up.

                                                        1. 2

                                                          If you want privacy should probably be using a VPN on foreign networks.

                                                          Restrictive networks need to become the new norm now. Allowing strangers on your network to spew DNS is asking for problems because this is the type of crap that infected machines do. I don’t need to permit infected gear on my networks sending thousands of pps of DNS traffic all because some people might have taken bad advice and hardcode DNS servers on their workstations/laptops. Catering to people taking bad advice on the internet should no longer be acceptable.

                                                          Sane traffic allowed out:

                                                          • HTTP
                                                          • HTTPS
                                                          • IPSEC
                                                          • OpenVPN

                                                          Nothing else. You use the internal NTP, DNS servers (which do use dnscrypt for its upstream), etc.

                                                          1. 4

                                                            If you want privacy should probably be using a VPN on foreign networks.

                                                            This is also advice we need to be careful with, because it’s usually really difficult to tell whether public VPN services are run by bad actors or not. You can never remove the need to trust a network altogether with a VPN, you just shift that need onto a different network. The average VPN user likely does not realise that.

                                                            Restrictive networks need to become the new norm now.

                                                            There is a time and a place for restrictive networks.

                                                            1. 2

                                                              Nothing else? What about SSH? SMTPS? IMAPS and POP3S? Are you suggesting that checking your email should be disallowed on most networks?

                                                              1. 1

                                                                Yes. None of those legacy mail protocols support 2FA and are frequently attacked by botnets because it helps evade IP rate limits while still executing their dictionary attacks.

                                                                End users don’t need SSH. Those that do should be smart enough to have a VPN.

                                                              2. 1

                                                                So infected machines tunnel over HTTP(S). Now you are relying on an HTTP specific firewall?

                                                                1. 1

                                                                  That’s fine. They can be infected and backdoored, but they won’t be spewing thousands of PPS of UDP and it’s very easy to deal with bad actors attempting to spam SYNs. It’s rather hard to DDOS TCP in comparison

                                                                  1. 1

                                                                    Setup two DNS servers; one inside the firewall the other outside. Firewall rules only permit DNS traffic between inside and outside DNS server. Intranet nodes can only query the inside DNS. Internet nodes can only spam the outside DNS.

                                                                    Blacklist IPs that spam the outside DNS. If DDoS is active, only serve requests from the Intranet, rely on the cache. Alternatively, only accept requests/responses from whitelisted DNS servers.

                                                          1. 2

                                                            I recently found that IPv6 HTTP traffic was unable to flow to my OpenBSD bytemark VPS (on a /56 netmask) unless I allow ICMP6 through the packet filter.

                                                            Perhaps someone knows why that might be?

                                                            FWIW, the bytemark docs are here: https://docs.bytemark.co.uk/article/finding-your-ipv6-address/

                                                            1. 5

                                                              And just as I posted this, I received an email from Bytemark saying that they’d updated their docs in light of my support request. The docs now say:

                                                              The role of ICMP has changed a little for IPv6. If your firewall has a default policy of deny then you may struggle to get traffic to or from your server without allowing traffic for certain ICMPv6 types. Types 1 – destination unreachable, 2 – Packet too big, 3 – Time exceeded and 4 – Parameter problem, for reporting errors to other devices. Types 128 – echo request, 129 – echo reply, for testing connectivity. Types 133 – Router Solicitation, 134 – Router Advertisement, 135 – Neighbor Solicitation, 136 – Neighbor Advertisement, for neighbour discovery. More information can be found at Wikipedia.

                                                              So if you are having routing/visibility problems with IPv6, then ICMP might be your problem!

                                                              Good show Bytemark.

                                                              1. 2
                                                              2. 5

                                                                ICMP6 takes the role of ARP (neighbor discovery). If you block it, you’ve just removed the ability for any other v6 nodes on the local network from being able to see you.

                                                                1. 1

                                                                  I’ve done some experimenting with determining the maximum MTU in IPv6 for an UDP application and if your host doesn’t process ICMPv6 Packet Too Big messages you’ll never be able to learn how to reach the other side in cases where your packets are too big (since v6 routers don’t fragment). These packets are effectively black holed.

                                                                1. 2

                                                                  I use Cisco WebEx Teams (formerly Cisco Spark) at work so I’ve been able to see the disaster first hand. The entire chat service was hard down for the first 24 hours with no ETA of resolution. When the service did come back up it still had small outages during business hours all week. Chatroom history is slowly being restored and many chatrooms are entirely glitched, unable to receive new messages or display old ones. Can you imagine the internet meltdown that would happen if Slack had this kind of outage? I feel bad for the ops employee: not only did they kill all the servers but they’ve almost certainly killed the business prospects of this giant, corporate, multi-year program with hundreds of contributors.

                                                                  1. 1

                                                                    OPS employee? I bet it was a developer.

                                                                    edit: I’ve seen the other side of this corporate curtain

                                                                  1. 3

                                                                    Note that the sources are available on github.com/GPGTools and you can always build it yourself.

                                                                    They offer support and binary releases and in exchange for that you pay something like 24 $. Considering this is on macOS, this might actually seem like a bargain.

                                                                    I am not sure if they will win a lot of people willing to pay but to be fair, they announced the change a long time ago. And as someone how does make a living with building software, I hope they can find a sustainable way to continue.

                                                                    1. 3

                                                                      Someone should get this into Macports or Brew then, so power users have a much easier way of installing it. I paid them instantly and have been waiting for this to happen. I believe this is the first time people have been able to upgrade MacOS day of release and have a compatible GPG plugin for Mail.app.

                                                                      1. 2

                                                                        I am unusually broke at the moment because I travelled a lot this year, but in general I am absolutely happy to pay for good software, and even more so if it’s open source. Examples: Textual is an amazing IRC client, it’s open source (you can compile it yourself for free) but it costs $7 to purchase (which is an absolute bargain).

                                                                        I think for now I’ll just compile my GPG Suite myself, but I will probably pay for it in the future.

                                                                      1. 3

                                                                        This has been a long time coming, but it’s unfortunately incomplete…

                                                                        2018-09-17

                                                                        New document that describes the data structures used for read-only access to Apple File System on unencrypted, non-Fusion storage.

                                                                        1. 2

                                                                          What does your diet mainly consist of?

                                                                          Depends on whether or not it is an active day or not…. If it’s a day of low physical activity (like in the office):

                                                                          • 1 slice of bread with peanut butter and 2 cups of tea for breakfast (I can’t eat much in the morning). Also some extra vitamin supplements as I don’t get enough vitamin D because of the lack of sunlight.
                                                                          • Lunch: Mostly some cucumber, lettuce and bell pepper and a hard boiled egg prepared at home while having breakfast. Drinks: Tea, water, milk or some fresh-fruit juice I can get at work. (yes, no coffee at all).
                                                                          • Snacks: Free fruit from the office or nothing at all.
                                                                          • Dinner: Whatever I decide to make that day, but I try to leave out as much of the the “dead calories” (potatoes, rice, noodles, etc.) as possible. So that’s mostly meat or fish and veggies. I don’t do desserts.

                                                                          Do you normally plan meals ahead or pickup food from places often?

                                                                          If it’s a day with high physical activity, like cycling to work (adds 2 hours of activity to my day), walking around a lot or doing sports, my diet is mostly the same as on a low activity day, but I take some extra snacks, take-outs or I might add potatoes, rice or something else I usually leave out to my dinner. Usually the choice is dictated by the “whatever I feel like”-heuristic, but only when my body “asks” for extra food twice.

                                                                          Has working out or being active pushed you towards a certain type of diet?

                                                                          No, although I eat considerably more when I am active.

                                                                          I have basically a few diet-rules I go by:

                                                                          • I try to avoid eating anything pre-processed or pre-packaged as much as possible, so I prepare as much of my own food by myself. It’s cheaper and it gives you control of portion sizes.
                                                                          • Only eat extra when my body “demands” it. So I ignore the first “hunger itch”. It’s unpleasant, but you’ll get used to it after about 6 weeks and by then it’s easy. However if there is a second wave of “hunger” (the type I can’t ignore and makes me cranky) I give in to it and eat an early lunch or dinner, or take some extra on the go.
                                                                          • Coffee contains fat and fat is calories (no wonder it gives you energy), a bottle of beer is 2 slices of bread (so a double breakfast), soft drinks are (full of sugar), (passive) smoking is bad and in places where smokers eat, the food is bad because you can’t taste properly and not being able to taste properly makes it easier to keep eating. Avoid all when possible.
                                                                          • Most important one of all: Always make portions on the small side. You don’t have to feel “full”, you just have to feel “not hungry” for the next few hours.

                                                                          What did push me towards this diet is that one day I just “felt heavy” and less comfortable moving around. Due to that I checked my BMI and it was 25,5. That was when I thought: “No more, I don’t want to feel like this ever again, it has to go down to at most 23, that means losing about 10 kg and keeping it of! I want don’t want to let my partner down and I want to set a good example when I get kids!”. That was the strongest motivator to push me towards this diet.

                                                                          1. 4

                                                                            Coffee contains fat and fat is calories (no wonder it gives you energy)

                                                                            Coffee has no fat. Espresso is ~ 1 calorie per oz

                                                                            1. 1

                                                                              Allright I oversimplified things. Let me rectify that.

                                                                              It’s true that coffee has no fat and that Espresso only has 1 kcal per cup. However, this only holds if you purely disregard what happens inside your body after you consume the coffee.

                                                                              All coffee except for old style paper-filtered coffee, does contain Cafestol which is an amino-acid which gets turned into fat and increases LDL-cholesterol significantly. So you have to drink only black filtered coffee without milk or sugar if you want the “coffee contains no fat”-statement to hold. Also: Milk contains fat, cafestol and suger both get turned into glucose and fructose, and glucose and fructose eventually get burned or turned into lipids which eventually are truned into (body) fat.

                                                                              But you don’t have to believe me blindly, you can actually do this very simple science experiment by yourself for less about 10€/$: Get a few test tubes, some distilled water and some ethanol with a >70% purity. Put about 1 ml of coffee into the test tube, add about 2 ml of ethanol and 2-3 ml of distilled water. Shake the tube so that everything mixes properly and leave it to rest for about an hour.

                                                                              If there is a dark band floating on top, that is the fat in you coffee.

                                                                              1. 1

                                                                                Coffee is coffee. If you add anything else to coffee grounds besides water, it’s no longer just “coffee”. I thought that was pretty obvious.

                                                                                I cannot find any paper that supports your claim that “cofestol” turns into fat. I can only find that it’s fat soluble, which is not the same thing. I can find that it stimulates insulin secretion and glucose uptake, but that is also not the same as your claim. If it turned into any form of sugar why would they be testing the use of cafestol to treat or prevent diabetes? That doesn’t make sense.

                                                                                1. 1

                                                                                  There are multiple studies, conducted from 1991 until 2015 by the Netherlands National Institute for Public Health and the Environment, sadly most of them are not (yet) translated into English, which confirm that cafestol lowers the production of bile acid, which in turn heightens levels of LDL-Cholesterol. So more fat from other sources stay fat and it stays in your bloodstream.

                                                                                  For the breakdown path of cafestol (one of many lipids) you simply have to look up the common breakdown-paths of lipids in literature about diabetes. You have to connect the dots yourself though, but this is why patients with type-2 diabetes should consume at most 4 cups of regular black coffee per day, unless they have a low blood-sugar.

                                                                                  If it turned into any form of sugar why would they be testing the use of cafestol to treat or prevent diabetes? That doesn’t make sense.

                                                                                  Here I can only speculate, because I just don’t know the angle of attack this research might take.

                                                                                  Given that type-2 diabetes is essentially insulin resistance and the fact some studies have shown that cafestol increases insulin production and glucose uptake in muscle tissue in rats, that might be why there is research into it.

                                                                                  But given that any diabetes treatment is basically “keep blood-sugar levels between this safe lower bound and this safe upper bound” and that cafestol has been shown to lower the production of bile acid (in humans), which is required to turn fat (also a lipid) into glucose, then it might be possible to lower the amount of new glucose from being formed by lowering the amount of lipids being converted into glucose. This in turn lowers the required amount of insulin, but it increases the amount of LDL-cholesterol in the patient’s blood. The underlying reasoning would be: We have to get rid of the excess lipids, just store them anywhere we can so we can prevent them from being metabolized into glucose. It’s no real solution and you’ll die of Atherosclerosis in 2 to 15 years, but you’ll die in a few weeks due to a too high blood sugar..

                                                                                  In the end I just don’t know, because this is still unpublished research.

                                                                                  I do know that I am sceptical as hell when I see texts with “too good to be true” statements like “Coffee decreases chances of getting diabetes!” and “Researchers are using coffee to treat diabetes!”, because it’s too simplistic, it’s what many people wish for and it aligns perfectly with certain corporate interests. “Extraordinary claims need extraordinary evidence” and history has shown us time and time again that these types of claims have failed to provide said extraordinary evidence, or worse; they turned out downright false.

                                                                                  1. 1

                                                                                    Thanks for the incredibly thoughtful response. I’ll have to do some more research on this.

                                                                                    1. 1

                                                                                      You’re welcome. I understand where you’re coming from, as I myself was also quite baffled and couldn’t believe it when I discovered this information for the first time. That’s why I did more research and devised a simple experiment so I could see for myself etc..

                                                                            2. 2

                                                                              Coffee suppresses the feeling of “tiredness”, no it doesn’t give you energy and no it doesn’t have fat.

                                                                              A double cream caramel frappucino with extra special pumpkin syrup from Starbucks is not “coffee”, it just contains it.

                                                                            1. 9

                                                                              Contrary to the comments at Reddit, I’m pretty sure Apple cannot do this unless you have installed a MDM profile…

                                                                              Locking, remote wipe, etc are limited to your iCloud account. There is no equivalent to “Google Play Services”. APNS has no control; it only handles push notifications.

                                                                              1. 15

                                                                                Contrary to the comments at Reddit, I’m pretty sure Apple cannot do this unless you have installed a MDM profile…

                                                                                When the OS is closed source how would you know?

                                                                                1. 12

                                                                                  If you think Apple has a gaping backdoor in all of their phones which violates the mission of their product line, then please prove me wrong. In fact, take this opportunity to short their stock and prove it to the world. You could make yourself really rich really fast.

                                                                                  Nobody else has done it, and everything Apple has done with their product line has been to constantly increase user security, not install backdoors for remote control and spying.

                                                                                  I do not think they are perfect, but this would be a huge blow to their public perception and would certainly tarnish their brand for years to come.

                                                                                  1. 7

                                                                                    Objectively, I think that u/user545 has a valid point. When proprietary software is in place there is no way to verify that such software does what the user expects it to do, and nothing more. Just because Apple has said it doesn’t spy on its users, doesn’t mean such a statement is true; and we cannot trust them, because we don’t know what the program does in the inside.

                                                                                    1. 9

                                                                                      Perhaps it’s not as severe as user545 says.

                                                                                      I think the argument can be transposed to anything done by anyone else:

                                                                                      • I didn’t see how cars were built. So I have to assume the worst.
                                                                                      • I didn’t see how roads were built. So I have to assume the worst.
                                                                                      • I didn’t audit this open source project’s source code myself. So I have to assume the worst.
                                                                                        • Or I only heard from someone that this source code checks out. But I don’t know that person, so I have to assume the worst (that they’re lying to me).
                                                                                        • I didn’t audit the crypto algorithms. So I have to assume the worst.
                                                                                        • I didn’t compile it myself. So I have to assume the worst.
                                                                                        • I didn’t compile my compiler myself. So I have to assume the worst.
                                                                                        • I didn’t compile my operating system myself with my own compiler. So I have to assume the worst.
                                                                                        • I didn’t mine and process the raw resources to create my computer. So I have to assume the worst.

                                                                                      Sure I can assume the worst, but then I probably wouldn’t live in a society.

                                                                                      “Assume the worst” feels like an impractical rule to follow. Instead, it’s a practical tradeoff of efficiency (of my time) and likelihood I need to “assume the worst”. I’m not discounting the valuable effort that security researchers do to audit and break into these systems. Especially if they take this approach, that’s great. But they’re way more qualified and have more resources (eg - time, money) than me to do it. I’m not going to blindly assume the worst that these security researchers are out to trick me.

                                                                                      I agree with feld. Apple isn’t perfect. They may change in the future. But Apple seem less likely than Google to implement a backdoor like this based on the way they position themselves in the market right now.

                                                                                      1. 5

                                                                                        You’re missing two things:

                                                                                        1. “They’re usually defective since suppliers dont care or have liability.”

                                                                                        2. “Intelligence agencies and law enforcement are threatening fines or jail for not putting secret backdoors in. The coercive groups also have legal immunity. Their targets can do 15 years if they talk.”

                                                                                        No 1 also applies to FOSS. With those premises, I definitely cant trust closed-source software to not have incidental or intentional vulnerabilities. Now, we’re back to thorough design and review by parties we trust. Multiple, skilled, mutually-suspicious groups.

                                                                                        1. 2

                                                                                          Thanks,

                                                                                          I agree with you on #1, including that it applies to FOSS. I may argue that a supplier has more incentive to fix it if you’re a potentially influential customer over a FOSS that has a disinterested maintainer (making you fall back to build-it-yourself or audit yourself. And to be clear, FOSS is definitely a better option than if the non-cooperative supplier is a monopoly). But I’d admit only be able to back up anecdotally, which isn’t a strong case.

                                                                                          For #2, couldn’t that also apply to key maintainers in FOSS if they are contributing to the same project? I’d take a random guess that governments may find it impossible to coerce a small set of individuals. 15 years would equality scare FOSS maintainers as well. Sure, a geographical barrier may make that more difficult, but I’d guess that human-based intelligence agencies like the CIA probably have some related experience in this. I agree that FOSS makes it harder to sneak one by reviewers, but maybe there’s not many people needed to coerce to get the backdoor in a release.

                                                                                          I only tangentially review security topics, so I’m not sure if that’s a realistic threat or just a tinfoil haty thought <:-).

                                                                                          I guess I’m putting more emphasis from the perspective of typical (non-technical) user of software to:

                                                                                          1. care more about security / privacy
                                                                                          2. pressure companies they support to have better security/privacy practices

                                                                                          Over distrusting all companies and have a significantly worse user experience of using software in general. Non-technical users generally like the fallback of technical support over just “figure it out yourself” or “you lost all your data because you couldn’t manage your secrets”.

                                                                                          I’m curious, if a company allowed you to audit their source code before you approved/used it, would that significantly minimize the advantages FOSS software have over proprietary software for you?

                                                                                          1. 2

                                                                                            I may argue that a supplier has more incentive to fix it if you’re a potentially influential customer over a FOSS that has a disinterested maintainer

                                                                                            This hasn’t been the case at all in the mobile space. The supplier has an incentive to not fix things so you buy a new device where as FOSS maintainers want your device to last as long as possible.

                                                                                            1. 2

                                                                                              I’d agree the motivation for some suppliers to upsell to newer devices, although I don’t really understand motivation for FOSS maintainers to want you to use your device as long as possible. As a one who maintained iOS libraries, there’s strong motivation to deprecate older devices/platforms since it’s a maintenance burden that sometimes hinders new feature work (and typically the most active contributors use the latest stuff). And when pitted against supporting the latest devices vs the older devices, chances are the newer stuff will win in those debates.

                                                                                              Thinking through the supplier stuff a bit more doesn’t make that much difference though. Sure, it doesn’t feel like a great business practice for a company to upsell. But it’s also how those companies stay in business. It could be viewed similarly to a maintenance support fee for existing devices. If suppliers offered the a retainer fee, it would effectively be the same thing then?

                                                                                              1. 2

                                                                                                The lineageOS team does amazing work keeping old Android devices on the latest release. Also means app devs don’t have to worry because these old devices support all the new apis and features.

                                                                                            2. 2

                                                                                              “For #2, couldn’t that also apply to key maintainers in FOSS if they are contributing to the same project?”

                                                                                              That’s a great observation. I held off mentioning it since people often say, “That’s speculation or conspiracy. Prove it with examples.” And the examples would have secrecy orders so… I just dropped the examples where they can find proof it happened. There very well could be coercive action against FOSS maintainers. Both Truecrypt developers and someone doing crypto on Linux filesystems kind of disappeared out of nowhere not talking about the project any longer. Now we’re into heresay and guesswork, though. Also, they might be able to SIGINT FOSS with a secrecy order. We might be able to counter that having people in foreign countries looking for the problem, submitting a fix, and the rule is to always take a fix. They have to spot the problem that might be out of their domain expertise, though.

                                                                                              Plenty of possibilities. I just don’t have anything concrete on mandated, FOSS subversion. I will say one of the reasons I’d never publish crypto under my own name or take money for it is this threat. I think it’s very realistic. I think we haven’t seen it play out since the popular libraries for crypto were so buggy that they didn’t need such a setup. If they did, they’d use it sparingly. Those also ran on systems that were themselves ridden with preventable 0-days.

                                                                                              Far as open vs closed with review, I wrote an essay on that here.

                                                                                              1. 2

                                                                                                Thanks for that essay, that was insightful.

                                                                                                I’m roughly remember the Truecrypt incident and that was suspect, although never came across the linux file system crypto circumstance. Was it similar to Truecrypt? Was that developer already known. My googling didn’t seem to show up any mention of that at all.

                                                                                            3. 1

                                                                                              There is one thing I am wondering about. Government agencies require backdoors but I would think they also require backdoors that are kept secret. How does that work with FOSS software? Alright yes they could sneek it in the compiled version maybe but distros are all moving to reproducible builds so that would be detected.

                                                                                              1. 2

                                                                                                Ignore the Karger/Thompdon attack: only happened twice that I know of. The nation-state attackers will go for low-hanging fruit like other black hats. They also need deniability. So, they’re most likely to either (a) use all bug hunting tools to find what’s already there and (b) introduce the kinds of defects people already do by accident. With (b), discoveries might not even burn the source if they otherwise do good work.

                                                                                                For FOSS, they’ll slip the vulnerability into a worthwhile contribution. It can be either in that component or be an interaction between it and others. Error-handling code of a complex component is a particularly-good spot since they often have errors.

                                                                                        2. 11

                                                                                          They are able to push updates over the internet and the whole thing is proprietary. I am unable to tell you what the system does because I cant see it. And at any time apple can push arbitrary code which could add a back door without anyone knowing.

                                                                                          When you can’t see what is going on you have to assume the worst.

                                                                                          1. 5

                                                                                            I can’t tell whether this is 1. a defense of open-source in general and android in particular or 2. a critique of apple.

                                                                                            Neither works.

                                                                                            1. See example of what just happened. or the firefox/mr robot partnership recently. open source does not automatically confer transparent privacy.

                                                                                            2. Apple has, in fact, emerged as a staunch defender of user privacy. There are many many examples of apple defending users against law enforcement.

                                                                                            You can’t wish Apple to be terrible about privacy and use that as the argument.

                                                                                            1. 3

                                                                                              Sure you can. They could take money to secretly backdoor the phone for NSA and use lawyers to tell FBI to get loss for image reasons. The better image on privcy leads to more sales. The deal with NSA puts upper bound on what FBI will do to them since they might just get data from NSA.

                                                                                              If that sounds far fetched, remember two things:

                                                                                              1. The telecoms were taking around $100 million each from NSA to give them data that they sometimes passed onto feds to use with parallel construction. Publicly they said they gave it out only with warrants. RSA went further to say they encrypted the data but weakened the crypto for $30 mil. The Core Secrets leak also said FBI could “compel” this.

                                                                                              2. In Lavabit trial, Feds argued he wouldnt have losses if customers didnt know he gave Feds the master key. He was supposed to do it under court order and then lie about it.

                                                                                              Given those two, I dont trust any profit-motivated company in US to not hand over data. Except maybe Lavabit in the past. Any of them could be doing it in secret for money that they take or get fines/jail.

                                                                                              1. 3

                                                                                                I would say Apple is more comparable to Lavabit than the others – they’re actively and publicly taking steps to protect their users’ privacy.

                                                                                                I wouldn’t argue that they will never do it, but to paint Apple and Google with the same brush on user privacy is silly and irresponsible.

                                                                                                1. 2

                                                                                                  Well, we know that the secret, court meeting was going to put him in contempt or else. He had to shut the business down to avoid it. Apple may have been able to do more due to both size and making case public debate. Then again, that may have been a one-time victory followed by a secret loss. You can’t know if there’s two legal systems in operation side by side, one public and one secret. I assume the worst if the secret system is aggressively after something.

                                                                                                  “I wouldn’t argue that they will never do it, but to paint Apple and Google with the same brush on user privacy is silly and irresponsible.”

                                                                                                  I agree with this. Apple is a product company. Google is a full-on, surveillance company. Google is both riskier for their users now and more over time as they collect more which more parties get in various ways.

                                                                                              2. 3

                                                                                                I am not defending android at all. As you can see in the OP post android is absolutely horrible for privacy and control. I also agree that open source is not flawless of course but open source enables us to have the opportunity to inspect the programs we use (usually while contributing features) from what I understand the firerfox event was pushed through a beta/testing channel and not through the FF source. I would hope all linux distros have this feature turned off when packaging FF.

                                                                                                The OP comment was asking me to prove that Apple is able to change user settings over the network and I think that is an unreasonable statement to make when the software is closed source. I also mentioned that it is possible as apple is able to push new updates at any time with arbitrary code. So they have the capability of doing anything that is possible hardware wise.

                                                                                                1. 2

                                                                                                  Fair on your 2nd point of responding to the OP and I don’t know whether they have the capability. However, they seem, at least at the moment, disinterested in taking random liberties with their users’ privacy.

                                                                                                  1. 3

                                                                                                    disinterested in taking random liberties with their users’ privacy.

                                                                                                    I think that’s probably true but no one in this thread actually knows and one day its quite likely that the US government will force them to backdoor devices if they haven’t already.

                                                                                                2. 1

                                                                                                  Apple has, in fact, emerged as a staunch defender of user privacy.

                                                                                                  this has to be a joke

                                                                                                3. 1

                                                                                                  How do you know they are able to do that then?

                                                                                                  Because all system updates that got installed on my phone came only after I manually approved them. Unless I am not aware of some previously demonstrated capability this sounds like exactly the same kind of unsubstantiated argument you are arguing against.

                                                                                                  1. 1

                                                                                                    What criteria do you use for approving or denying updates and how would that be able to stop a backdoor being installed?

                                                                                                    1. 2

                                                                                                      It doesn’t matter since the original argument was that Apple can do the same thing (automatically install/change software on your device) which they cannot. You have to assent to the installation (of updates, backdoor or whatever). May not be a difference you care about, but I do.

                                                                                                      I agree that black box software makes it impossible to know if software can be trusted, but binary package of an open source software is also just a black box if I am not able to generate the same hash when compiling myself which in my admittedly not recent experience happened a lot.

                                                                                                      1. 1

                                                                                                        “You have to assent to the installation “

                                                                                                        You would need a copy of source for all priveleged hardware and software on their platform to even begin to prove that. You dont have that. So, you don’t know. You’re acting on faith in a profit-motivated, company’s promises.

                                                                                                        I’ll also add one that has enough money to do a secure rewrite or mod of their OS but doesnt intentionaly. They don’t care that much. They’re barely even investing into Mac OS X from what its users say. Whereas, Sun invested almost $300 million into redoing Solaris for version 10. That brought us things like ZFS.

                                                                                                        A company with around a $100 billion that cares less about QA than smaller businesses shouldnt be trusted at all. They’ve already signalled that wealth accumulation was more important.

                                                                                                        Meanwhile, tiny OK Labs cranked out mobile sandboxing good enough that General Dynamics bet piles of money on them for Defense use. Several other companies cranked out security-enhanced CPU’s, network stacks, DNS, end-to-end messaging, and so on. Quite a few were for sale, esp those nearing bankruptcy. Shows Apple had plenty of opportunities to do the same or buy them. Didnt care. They’ll make billions anyway.

                                                                                                        1. 2

                                                                                                          I agree with pretty much everything you say and while interesting, I am not sure how it is relevant to what I said.

                                                                                                          I did not argue that one should trust Apple (even though I do think iPhone has a better track record than Android). My point was simply that all other things being equal I prefer platforms that don’t suddenly change on some company’s whim and let me decide when or if I want to perform an update and that AFAICT Apple does not push those updates without user’s consent.

                                                                                                          I assume your argument is that consenting is meaningless as I cannot perform any reasonable security analysis of what I will receive. True that I can’t, but I also value predictability and speaking from a personal experience I feel I lose some of it with auto-updates.

                                                                                                          1. 1

                                                                                                            I assume your argument is that consenting is meaningless as I cannot perform any reasonable security analysis of what I will receive. True that I can’t, but I also value predictability and speaking from a personal experience I feel I lose some of it with auto-updates.

                                                                                                            I think you are missing the point. Your iPhone has convinced you that it would only ever install an update if you approved it, but you have no way of knowing that there isn’t already a way for Apple to push software without your consent, in a way that you wouldn’t detect.

                                                                                                            I’m sure if you looked at the EULA that you agree to when you use an iPhone, Apple has every legal right to do this even if they try to create an image of a company that wouldn’t.

                                                                                              3. 4

                                                                                                objdump -d

                                                                                                1. 3

                                                                                                  When the OS is open source how would you know? Have you personally audited all of linux? How do you know you can trust third-party audits? I don’t think “it’s open source” provides much in terms of security all things considered.

                                                                                                2. 3

                                                                                                  how do you know, what APNS does.

                                                                                                1. 1

                                                                                                  Hmm, now the question is can I make my firewall drop fragments only for UDP port 53? Dropping all frags is a terrible thing to do…

                                                                                                  1. 1

                                                                                                    The IPv6 tunnels were fun for learning about IPv6 but probably should not have been allowed to exist as long as they have. Native or bust. MTU issues with IPv6 are a nightmare to deal with.

                                                                                                    1. 1

                                                                                                      We’d only get bust. Ipv6 was not designed for anything less than a flag day.

                                                                                                    1. 12

                                                                                                      Lots of null pointer dereferences, use-after-free, and double free. OpenBSD really needs a language with affine types or smart pointers that integrates with C. ;)

                                                                                                      1. 6

                                                                                                        Such a language needs to work on every hardware platform they support and have a BSD licensed compiler/toolchain 🙃

                                                                                                        1. 5

                                                                                                          I actually think starting to use C++ in kernel is no-brainer, like GCC did. C++ doesn’t have hardware or toolchain problem, does it?

                                                                                                          1. 1

                                                                                                            Although I’m against C++, it’s clearly an option with more safety features and low-cost abstractions all the time. I”ll note that folks developing L4 microkernels and Genode started using it for those reasons. At this point, I’d rather whatever it is be a safer C with better abstractions that outputs vanilla C. That would solve most of tooling and integration issues that come with language switch. It also dodges C++‘s huge complexity. It’s ridiculously complex.

                                                                                                            1. 3

                                                                                                              a safer C with better abstractions that outputs vanilla C.

                                                                                                              Sounds like Nim to me. MIT license.

                                                                                                              1. 3

                                                                                                                It’s close! I’m eyeballing it for that use with Brute-Force Assurance. It would have way more acceptance than a Scheme-based solution. I’d have to swap its syntax out since C developers switch to C-like languages more than Python-like languages. The compiler for this purpose should produce C that looks like what a person would write more than a machine. It should at least be an option. Lets it get used incrementally in existing, C projects. Finally, the people I see online griping about the compiler means they need to focus hard on getting it in good shape or someone has to build a separate, certifying compiler.

                                                                                                                So, that’s what I was thinking when I assessed Nim as C replacement in general and for safety critical. Oh yeah, contracts! Frama-C or Ada-style contracts supported by default. Lets you encode whatever extra stuff the type system doesn’t already handle. I don’t know if they have contracts.

                                                                                                          2. 2

                                                                                                            Im sure they could build the language or C extensions given they built a whole OS and maintained (still do?) a compiler for it. It would also help them achieve their security goals better than their developers are doing now with C language. A good investment I’d say.

                                                                                                            1. 10

                                                                                                              This could happen if one or more people with interest and motivation showed up and managed to work well with the project to integrate this with the system as yet another form of mitigation.

                                                                                                              As for the existing devs, they are all already very busy scratching their own itches and pursue their own ideas, some related to security, some not. And generally they don’t like to be told what to work on in the time they volunteer.

                                                                                                              1. 1

                                                                                                                Exactly. The average coder in Rust is currently outperforming the OpenBSD team on these kinds of bugs due to type system. That means these bugs happen since they don’t care enough to prevent them. They’re about QA and mitigation tech up to a certain point with certain bug-adding tech (eg C language). Past that point or with different mitigations (esp language), they start making excuses about time, itches, and so on. I’ll keep pointing this out every time evidence of easily-prevented bugs comes in. Maybe something will click in a reader’s head that leads to a solution.

                                                                                                                Many of them also tell other people how they should be doing UNIX design, quality or security. Sometimes even in a snooty way. They like doing that despite aggravation it might cause others. You say those same people don’t like “to be told” they should use more secure tech in a security-focused project. It sounds like there’s a life lesson in there somewhere on top of some security lessons.

                                                                                                                1. 5

                                                                                                                  since they don’t care enough to prevent them

                                                                                                                  That’s a tad inflammatory nay? Suggesting that not using rust in tantamount to not caring. Its not like the Linux/BSD kernel could be rewritten in rust in a day, there is 20+ years of development in there.

                                                                                                                  And while its not exactly a fair compairson as its been run against linux for longer, 9 issues (which have been fixed) versus quite a few in linux suggests something in OpenBSD is working.

                                                                                                                  1. 1

                                                                                                                    Yeah, a tad inflammatory to match the style of their mailing lists talking about other OS’s or hardware vendors not doing enough for security. I always give them credit for their strong points of simplified UNIX, code review/quality, mitigations, and great documentation. Plus, I like a few of them personally.

                                                                                                                    Far as your counterpoint, it’s a strawman (full rewrite) that’s not even what Im proposing. I’m saying folks that cared seeing the language cause issues would make a safer version like others did in other projects (eg Clay, Cyclone). One highly-compatible with C. They’d write new code in that language. The extensive rewrites of existing code they already do would be done in that language. Over time (years), most or all the OS would be converted to the safer language. Someone might even write tools to automate this.

                                                                                                                    1. 5

                                                                                                                      The idea of a slightly modified C which would somehow prevent use-after-free and similar bugs is good. It’s similar to other ideas OpenBSD has already realized such as adding C API functions which are easier to use safely, or hardening of the C run-time against ROP. And it’s not as if the C we’re writing did not contain non-standard extensions already (packed structs, gcc-isms inherited by clang, etc.)

                                                                                                                      Now, where are some compiler-writing C langauge lawyer academics with the needed skills who would sit down with a bunch of OpenBSD hackers and volunteer a lot of their spare time for this? In over 10 years of involvement with the project I’ve never met a person with this skill set. In a volunteer project you have to work with the skills you happen to get.

                                                                                                                      1. 1

                                                                                                                        Glad you’re open to the possibility if you had help for it. The people behind Clay and Cyclone might have helped given they were already doing hardest parts. It’s possible you didn’t know those languages exist. The folks good at researching and developing languages usually aren’t good at polish, outreach, and so on.

                                                                                                                        It’s possible we need a sponsor organization or new type of volunteer for such a role. One that’s a middle-person between the team with time to build compilers and the people that would use them. Such a person would need to be able to influence compiler developers to ensure they don’t do anything that kills adoption. I figure there’d be a lot of negotiations with middle person doing tie breakers on stuff people were divided on. Probably also need to be a compiler developer themselves so they can do the polish, packaging, and later maintenance.

                                                                                                            2. 2

                                                                                                              I realize this is mostly bikeshedding, but does the core team regularly (or ever) consider this? Or is this seen as too much overhead - learning the subtleties of a new language/implementation on top of the difficulty of os/kernel development. I would think the D language folks would love to team up with one of the BSDs to focus on whatever language demands the OS team would come up with.

                                                                                                          1. 2

                                                                                                            Yet another reason to try for BSD jails and ansible.

                                                                                                            1. 2

                                                                                                              If only any of the BSDs had an init system with declarable units, instead of the hack that is shell scripts.

                                                                                                              1. 1

                                                                                                                Nobody is preventing you from installing and using one

                                                                                                                1. 2

                                                                                                                  Yes, and nobody is preventing me from using Linux with systemd either, which I rather do until they fix this. If they never fix it, that’s fine too.

                                                                                                                  1. 1

                                                                                                                    How many service units are you writing on a daily basis that makes Systemd a necessity for your use case? Do Linux packages typically ship without service units and force you to do it yourself?

                                                                                                                    1. 1

                                                                                                                      Well, none of the Fun parts even come from the official repos. Plus there’s of course all internally developed stuff – somebody needs to write init scripts or unit files for those. Getting a unit file 95% correct on the first try is possible.

                                                                                                                      You may be right that systemd is not necessary for anything I do. It’s just a whole lot more convenient than the alternatives.

                                                                                                                      1. 1

                                                                                                                        That’s the difference with FreeBSD: we don’t have a small “official” repo. Our community maintained ports tree is huge and everything that needs an rc script comes with one.

                                                                                                                        1. 1

                                                                                                                          My frequent experience from FreeBSD has been that not all of those rc scripts in the ports tree work 100% well. Systemd units tend to work pretty solidly, on the other hand.

                                                                                                                          1. 1

                                                                                                                            If you happen to remember any specific poor experiences please send me a message… I’ll hunt down the rc scripts and fix them.

                                                                                                            1. 2

                                                                                                              It occurs to me that if you do not have the right to benchmark then you do not have the right to test that the product works as advertised. This cannot be legal.

                                                                                                              1. 2

                                                                                                                This license forbids systems integrators from publishing benchmarks related to this microcode. Presumably because Intel reserves that right to themselves. If you are not a systems integrator it doesn’t apply to you. If you are a systems integrator not only can you benchmark, clause 4 makes it clear you are under no obligation to share those results, even with Intel.

                                                                                                                1. 1

                                                                                                                  We don’t want to get submissions for every CVE and, if we do get CVEs, we probably want them tagged security.

                                                                                                                  1. 16

                                                                                                                    while I agree with you in this case, I don’t particularly like the “I speak for everyone” stance you seem to be taking here.

                                                                                                                    1. 9

                                                                                                                      This one is somewhat notable for being the first (?) RCE in Rust, a very safety-focused language. However, the CVE entry itself is almost useless, and the previously-linked blog post (mentioned by @Freaky) is a much better article to link and discuss.

                                                                                                                      1. 4

                                                                                                                        Second. There was a security vulnerability affecting rustdoc plugins.

                                                                                                                    2. 4

                                                                                                                      Do you think an additional CVE tag would make sense? Given there’s upvotes some people seem to be interested.

                                                                                                                      1. 2

                                                                                                                        That’d be a good meta tag proposal thread.

                                                                                                                      2. 4

                                                                                                                        Yeah, I’d rather not have them at all. Maybe a detailed, tech write-up of discovery, implementation, and mitigation of new classes of vulnerability with wide impact. Meltdown/Spectre or Return-oriented Programming are examples. Then, we see only the deep stuff with vulnerability-listing sites having the regular stuff for people using that stuff.

                                                                                                                        1. 5

                                                                                                                          seems like a CVE especially arbitrary code execution is worth posting. my 2 cents

                                                                                                                          1. 5

                                                                                                                            There are a lot of potentially-RCE bugs (type confusion, use after free, buffer overflow write), if there was a lobsters thread for each of them, there’d be no room for anything else.

                                                                                                                            Here’s a list a short from the past year or two, from one source: https://bugs.chromium.org/p/oss-fuzz/issues/list?can=1&q=Type%3DBug-Security+label%3AStability-Memory-AddressSanitizer&sort=-modified&colspec=ID+Type+Component+Status+Library+Reported+Owner+Summary+Modified&cells=ids

                                                                                                                            1. 2

                                                                                                                              i’m fully aware of that. What I was commenting on was Rust having one of these RCE-type bugs, which, to me, is worthy of discussion. I think its weird to police these like their some kind of existential threat to the community, especially given how much enlightenment can be gained by discussion of their individual circumstances.

                                                                                                                              1. -2

                                                                                                                                But that’s not Rust, the perfect language that is supposed to save the world from security vulnerabilities.

                                                                                                                                1. 4

                                                                                                                                  Rust is not and never claimed to be perfect. On the other hand, Rust is and claims to be better than C++ with respect to security vulnerabilities.

                                                                                                                                  1. 0

                                                                                                                                    It claims few things - from the rustlang website:

                                                                                                                                    Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety.

                                                                                                                                    None of those claims are really true.

                                                                                                                                    It’s clearly not fast enough if you need unsafe to get real performance - which is the reason this cve was possible.

                                                                                                                                    It’s clearly not preventing segfaults - which this cve shows.

                                                                                                                                    It also can’t prevent deadlocks so it is not guaranteeing thread safety.

                                                                                                                                    I like rustlang but the claims it makes are mostly incorrect or overblown.

                                                                                                                                    1. 2

                                                                                                                                      Unsafe Rust is part of Rust. I grant you that “safe Rust is blazingly fast” may not be “really true”.

                                                                                                                                      Rust prevents segfaults. It just does not prevent all segfaults. For example, a DOM fuzzer was run on Chrome and Firefox and found segfaults, but the same fuzzer run for the same time on Servo found none.

                                                                                                                                      I grant you on deadlocks. But “Rust prevents data race” is true.

                                                                                                                                  2. 2

                                                                                                                                    I’m just going to link my previous commentary: https://lobste.rs/s/7b0gab/how_rust_s_standard_library_was#c_njpoza