1. 4

    I’m Chris and I also have a technical blog where I share my experiences, open source projects and sometimes opinions about “Devops”, Automation, Open Source, Rust, Golang, Security and alike.

    Link: https://chr4.org

    1. 1

      If I understand the post correctly, this seems like a too big obvious failure. I kind of can’t believe Debian and Ubuntu never thought about that.

      Did someone try injecting a manipulated package? I’d assume that at least the signed manifest contains not only URLs and package version but also some kind of shasum at least?

      1. 2

        Looks like that’s exactly what apt is doing, it verifies the checksum served in the signed manifesto: https://wiki.debian.org/SecureApt#How_to_manually_check_for_package.27s_integrity

        The document mentions it uses MD5 though, maybe there’s a vector for collisions here, but it’s not as trivial as the post indicates, I’d say.

        Maybe there’s marketing behind it? Packagecloud offers repositories with TLS transport…

        1. 2

          Modern apt repos contain SHA256 sums of all the metadata files, signed by the Debian gpg key & each individual package metadata contains that package’s SHA256 sum.

          That said, they’re not wrong that serving apt repos over anything but https is inexcusable in the modern world.

          1. 2

            You must live on a planet where there are no users who live behind bad firewalls and MITM proxies that break HTTPS, because that’s why FreeBSD still doesn’t use HTTPS for … anything? I guess we have it for the website and SVN, but not for packages or portsnap.

            1. 1

              There’s nothing wrong with being able to use http if you have to: https should be the default however.

              1. 1

                https is very inconvenient to do on community run mirrors

                See also: clamav antivirus

                1. 1

                  In the modern world with letsencrypt it’s no where near as bad as it used to be though.

                  1. 1

                    I don’t think I would trust third parties to be able to issue certificates under my domain.

                    It is even more complicated for clamav where servers may be responding to many different domain names based on which pools they are in. You would need multiple wildcards.

            2. 1

              each individual package metadata contains that package’s SHA256 sum

              Is the shasum of every individual package not included in the (verified) manifesto? That would be a major issue then, as it can be forged alongside the package.

              But if it is, then forging packages should require SHA256 collisions, which should be safe. And package integrity verified.

              Obviously, serving via TLS won’t hurt security, but (given that letsencrypt is fairly young) depend on a centralized CA structure and additional costs - and arguably add a little more privacy on which packages you install.

              1. 3

                A few days ago I was searching about this same topic when after seeing the apt update log and found this site with some ideas about it https://whydoesaptnotusehttps.com, including the point about privacy.
                I think the point about intermetdiate cache proxys and use of bandwith for the distribution servers probably adds more than the cost of a TLS certificate (many offer alternative torrent files for the live cd to offload this cost).

                Also, the packagecloud article implies that serving over TLS removes the risk of MitM, but it just makes it harder, and without certificate pinning only a little. I’d defer mostly to the marketing approach on this article, there are call-to-action sprinkled on the text

                1. 1

                  https://whydoesaptnotusehttps.com

                  Good resource, sums it up pretty well!

                  Edit: Doesn’t answer the question about whether SHA256 sums for each individual package are included in the manifesto. But if not, all of this would make no sense, so I assume and hope so.

                  1. 2

                    Hi. I’m the author of the post – I strongly encourage everyone to use TLS.

                    SHA256 sums of the packages are included in the metadata, but this does nothing to prevent downgrade attacks, replay attacks, or freeze attacks.

                    I’ve submit a pull request to the source of “whydoesaptnotusehttps” to correct the content of the website, as it implies several incorrect things about the APT security model.

                    Please re-read my article and the linked academic paper. The solution to the bugs presented is to simply use TLS, always. There is no excuse not to.

                    1. 2

                      TLS is a good idea, but it’s not sufficient (I work on TUF). TUF is the consequence of this research, you can find other papers about repository security (as well as current integrations of TUF) on the website.

                      1. 1

                        Yep, TUF is great – I’ve read quite a bit about it. Is there an APT TUF transport? If not, it seems like the best APT users can do is use TLS and hope some will write apt-transport-tuf for now :)

                      2. 1

                        Thanks for the post and the research!

                        It’s not that easy to switch to https: A lot of repositories (incl. die official ones of Ubuntu) do not support https. Furthermore, most cloud providers proivide their own mirrors and caches. There’s no way to verify whether the whole “apt-chain” of package uploads, mirrors and caches is using https. Even if you enforce HTTPS, the described vectors (if I understood correctly) remain an issue in the mirrors/ cache scenario.

                        You may be right, that current mitingations for the said vectors are not sufficient, but I feel like a security model in package management that relies on TLS is simply not sufficient and the mitingation of the attack vectors you’ve found needs to be something else - e.g. signing and verifing the packages upon installation.

                  2. 2

                    Is the shasum of every individual package not included in the (verified) manifesto? That would be a major issue then, as it can be forged alongside the package.

                    Yes, there’s a chain of trust: the signature of each package is contained within the repo manifest file, which is ultimately signed by the Debian archive key. It’s a bit like a git archive - a chain of SHA256 sums of which only the final one needs to be signed to trust the whole.

                    There are issues with http downloads - eg it reveals which packages you download, so by inspecting the data flow an attacker could find out which packages you’ve downloaded and know which attacks would be likely to be successful - but package replacement on the wire isn’t one of them.

            1. 7

              Neat idea! One question though: How do you handle renewals? In my experience, postgresql (9.x at least) can only re-read the certificate upon a server restart, not upon mere reloads. Therefore, all connections are interrupted when the certificate is changed. With letsencrypt, this will happen more frequently - did you find a way around this?

              1. 5

                If you put nginx in front as a reverse TCP proxy, Postgres won’t need to know about TLS at all and nginx already has fancy reload capability.

                1. 3

                  I was thinking about that too - and it made me also wonder whether using OpenResty along with a judicious combination of stream-lua-nginx-module and lua-resty-letsencrypt might let you do the whole thing in nginx, including automatic AOT cert updates as well as fancy reloads, without postgres needing to know anything about it at all (even if some tweaking of resty-letsencrypt might be needed).

                  1. 1

                    That’s funny I was just talking to someone who was having problems with “reload” not picking up certificates in nginx. Can you confirm nginx doesn’t require a restart?

                    1. 1

                      Hmm, I wonder if they’re not sending the SIGHUP to the right process. It does work when configured correctly.

                  2. 2

                    I’ve run into this issue as well with PostgreSQL deployments using an internal CA that did short lived certs.

                    Does anyone know if the upstream PostgreSQL devs are aware of the issue?

                    1. 20

                      This is fixed in PG 10. “This allows SSL to be reconfigured without a server restart, by using pg_ctl reload, SELECT pg_reload_conf(), or sending a SIGHUP signal. However, reloading the SSL configuration does not work if the server’s SSL key requires a passphrase, as there is no way to re-prompt for the passphrase. The original configuration will apply for the life of the postmaster in that case.” from https://www.postgresql.org/docs/current/static/release-10.html

                  1. 2

                    If you are a fellow developer, you might find developermail.io being a fit. It’s configurable via Git and it’s supposed to be something like an “Heroku for email”.

                    Disclaimer: I’m one of the creators.

                    1. 2

                      Seems like an interesting project.

                      I would probably reach for wget (can read an a file of urls to fetch) or a combination of curl and xargs (or gnu parallel), before trying a bespoke tool like this though. That said, the X-Cache-Status statistics are neat, if you need that.

                      1. 2

                        That’s what I thought. When looping through a file with a few hundred thousand entries with bash/ curl, I had a throughput of ~16 requests/second, while cache_warmer easily was able to do >500 req/s.

                        Thanks for the hint, I should probably add that to the post.

                        1. 1

                          Indeed, looping via bash would be slow due to not reusing the connection. With a carefully crafted xargs, you should be able to get multiple urls on the same line (eg. curl url1 url2 url3...). Then curl /should/ reuse a connection in that case. If curl had a ‘read urls from a file’ parameter, it would be quite a bit easier to script, but alas it currently does not.

                      1. 5

                        I’ve found some more information (not from original sources though, so treat with care):

                        • HN thread of someone stepping down from TSC (with a lot of occurances of “SJW” in the comments)
                        • Someone in a german forum did some research and claims the actual reasons are “thoughtless use of pronouns” and “assumtions of gender”.

                        Personally, I’m kind of not sure whether it’s a good idea to give this too much attention. It feels like either side can only lose from this….

                        1. 6

                          The fact this is plausibly useful is a sad comment on the state of software engineering.

                          1. 2

                            How would you describe the alternative desired state? That insecure protocols don’t exist? That engineers would have deeper knowledge of cryptography?

                            1. 8

                              Distributions of major server software would come with good configurations out of the box, alleviating every developer from being responsible for configuring things.

                              https://caddyserver.com/ is a great example of this; you configure it to do what your app needs, all the TLS defaults are well curated.

                              1. 4

                                While I agree that a “reasonably secure default” should be standard, mostly you have to find a trade-off between security and compatibility. If you need support for IE8, there’s no way around SHA. If you want to support Windows XP or Android 2, there’s no hope at all. If you want it more secure (as of today) you fence out most Androids (but 4.x), Javas, IEs, Mobile Phones, non-up-to-date browsers. Unfortunately, there is no one size fits all.

                                1. 3

                                  On the other hand, compatibility with older software is very easy to figure out (people see an error message), whereas insecure configuration appears to work perfectly fine. I also believe developers are more likely to know that they need to support some obsolete software (modern web development doesn’t “just work” on IE8 or Android 2) than about the newest TLS configuration options.

                                  1. 2

                                    I think if you want that, we ought to have APIs that express things in terms of goals, instead of implementation details: ssl_ciphers +modern +ie8 maybe. Then it’s clear what needs to be changed to drop a platform, instead of it being a guessing game.

                                    1. 2

                                      This would be great. This is exactly what I’m trying to provide the user with the snippets in nginx.vim: Choose between ciphers-paranoid, ciphers-modern, ciphers-compat, ciphers-low cipher suites.

                            1. 4

                              First thought: didn’t I see this in the new nginx vim-syntax posted here a while ago?

                              Ah.. it’s by the same author.

                              Nice work, ch4r. It helped me get rid of a couple I had in my 2014 nginx ssl config. Hat tip.

                              1. 2

                                Good catch :)

                                sslsecure.vim is actually trying to get some of the security features from nginx.vim to work with other configuration files and source code as well.

                                Good to hear it actually helped you re-securing your config!

                              1. 2

                                FYI: I’ve added support for embedded syntax highlighting for ERB/ Jinja templates and LUA.

                                1. 3

                                  I’m currently working on https://developermail.io - an email-SaaS that is configured with git - might be referred to as the “Heroku for email”.

                                  I’d apprechiate if you consider adding it to your list!

                                  1. 3

                                    Oh cool! I love stuff like this - will bear in mind in case the need arises.

                                    Other than exposing the sieve rules over git, how do you compare to fastmail offering more-or-less the same featureset / price point?

                                    Your security page claims you use SHA-512 for password storage.

                                    My understanding of crypto is… not deep, but that looks like poor marketing given http://security.stackexchange.com/questions/52041/is-using-sha-512-for-storing-passwords-tolerable is one of the first things I came across trying to figure out if it was OK.

                                    1. 2

                                      The unique selling point would be the git configuration, including all the advantages that git brings: You know who changed what when, you can have review processes using branches/ pull-requests, rollback changes and do bulk changes more easily. Plus you can comment your configuration. Plus: It feels more natural and leet for developers :)

                                      Additionally, as you pointed out, the sieve control is way more powerful.

                                      1. [Comment removed by author]

                                        1. 2

                                          That’s a good thought. developermail uses rounds=100000, so this should probably be mentioned on the site.

                                          1. [Comment removed by author]

                                            1. 1

                                              From what I know, the used number of rounds can be treated similar to the salt. It’s not required to be hidden, as it’s stored alongside (or at least accessible with the same credentials/ from the same applications) in the system (in fact: CRYPT-SHA512 stores it in the form CRYPT-SHA512$6$rounds=xxx$HASH. In case of a data-breach security shouldn’t rely on the salt or whether the number of rounds are known.

                                              While I don’t agree with “security by obscurity”, I don’t want to escalate the discussion here.

                                              Thanks for your feedback!

                                      2. 2

                                        Regardig your observation for SHA-512:

                                        Obviously, one would want to have script or bcrypt, maybe even Argon2 instead of SHA-512. There’s however another factor when choosing the right algorithm in my opinion, and that’s the implementation.

                                        Let’s take dovecot as an example. According to the documentation SHA-512 is the strongest scheme implemented on all platforms (it mentions bcrypt but with the annotation that it’s not available on most Linux distributions). Furthermore, a Argon2/ scrypt plugin is mentioned, but it’s third party. Of course I’ve considered using one of the mentioned algorithms, do not feel competent enough to review a 3rd party plugin on my own regarding its implementation - Especially since dovecot itself was recently reviewed and received an extremly positive rating. A bad implementation of a secure algorithm may introduce other attack vectors or security issues. In case I missed something, I’d apprechiate feedback. And of course I’ll follow the ongoing implementation and new security features closely to improve the security whereever possible. I’m also wondering how other email providers handle the issue. Most of them are pretty silent on what algorithms they’re using from what I’ve observed. As anyone some insights to share?

                                        TL;DR: I’d love to use scrypt() but I’m not sure whether to trust the inofficial plugins implementing it.

                                        Edit: developermail.io uses rounds=100000. While one would still prefer scrypt(), this should increase computational requirements a lot. I’m going to add this to the website.

                                    1. 8

                                      For anyone considering this, there’s also a more elaborate fuzzy searcher called fzf that comes with shell history search shortcut

                                      1. 1

                                        I’m using fzf and I was wondering whether there’s any advantage using hstr (performance, packages, etc) - as it offers the same feature (and a lot more).

                                      1. 4

                                        The certificate for https://download.servo.org/ seems to have expired. Unfortunately, I can’t find checksums for the downloaded files elsewhere to verify the download :(

                                        Super exited to try out Servo, though!

                                        1. 1

                                          There’s an issue already on Github

                                        1. 1
                                          • Safety and security. I’m looking forward to Firefox with more and more Rust components.
                                          • C compatible. It’s possible to exchange more and more of your critical system components with Rust code. You can even write Kernel modules in Rust! This would eradicate so many attack vectors.
                                          • Plus: The community is incredibly friendly and helpful
                                          • And finally: It’s a Mozilla project. Mozilla is one of the few defenders and advocates of internet users left out there.
                                          1. 8

                                            Okay, that’s great and all.

                                            What have you, chr4, personally used Rust for?

                                            You didn’t answer any of the above questions. Concrete examples please.

                                            1. 6

                                              Well, I answered the question about the Rust features I apprechiate. But you’re right, I was missing out on the examples:

                                              I’m currently rewriting some of my C projects in Rust (with more or less success), as well as tinkering around with rewriting a Go JSON API. The latter is probably not very useful, as I think Go is the better fit for the job, but it helps me improving my Rust skills. Furthermore, I’m currently working on An Interpreter In Rust.

                                              All projects so far are not for commercial projects.

                                          1. 17

                                            This post lacks addressing the fundamental reasons of not using a public cloud. While it’s beneficial for tons of usecases (distributing and computing public content, like livestreams, having access to compute power for a few hours without having to buy tons of hardware, etc), it also has downsides:

                                            Security

                                            It’s a significant attack vector to share resources with others. See the Rowhammer vulnerability, or this paper (extracting private keys on shared CPU systems, there’re more papers, this was the first I found) for example.

                                            Vendor lock-in

                                            If you rely on e.g. AWS or GCE services that can’t be migrated easily (e.g. Route53), you’ve locked yourself in, and you can only switch providers if necessary with enourmous efforts.

                                            Privacy

                                            I think it’s just irresponsible to host private customer data in locations that don’t provide basic security you can trust. Your database drive might end up in the hands of a customer or third-party service contractor, let alone issues of government access. Encryption, e.g. on GCE, which relies on keys stored on their servers only address a few of those issues.

                                            Price

                                            Running a 24/7 machine on a major cloud provider usually costs a lot.

                                            Edit: Formatting

                                            1. 7

                                              Price

                                              Running a 24/7 machine on a major cloud provider usually costs a lot.

                                              This is especially true when you’re doing mainly compute, rather than hosting. Advocates of public clouds often correctly focus on the variety of infrastructure you’d have to handle yourself if you don’t go with them: uptime, several kinds of failover, fault-tolerant storage, load scaling, etc. Properly costed, that may well narrow or eliminate the price difference. But in data analysis it’s much more common to have a situation where I don’t need all that good uptime, failover, integration with databases, CDNs, etc. Instead I just need a lot of compute throughput over a period of months, for ideally as little money as possible.

                                              For example, if you’re training deep neural networks on GPUs, which is a very common current use-case both in academia and industry, AWS for a single month of usage will charge you about what it costs to buy the machine outright. Even if you add in substantial overhead, it’s hard to make AWS come out ahead here.

                                              1. 1

                                                If you rely on e.g. AWS or GCE services that can’t be migrated easily (e.g. Route53), you’ve locked yourself in, and you can only switch providers if necessary with enourmous efforts.

                                                Sad face - I have some domains in R53. Why is it so hard to migrate?

                                                1. 3

                                                  I suppose you can migrate away (with the usual pain of migrating DNS) unless you use “special features”.

                                                  I should’ve made it clearer in my comment: When saying “Lock-In”, a was refering to “special” features of services. If you’re running a virtual machine with PostgreSQL on a cloud provider, you can migrate to another provider and set up your virtual machine there. But if you use managed databases (like e.g. Amazon Redshift), migrating away won’t work as easily, as Redshift is not an OpenSource product you can just run anywhere.

                                                  1. 1

                                                    Got it. Thanks for clarifying. Yeah some of those extensions sure do have a warm embrace, huh.