1. 7

    Neat idea! One question though: How do you handle renewals? In my experience, postgresql (9.x at least) can only re-read the certificate upon a server restart, not upon mere reloads. Therefore, all connections are interrupted when the certificate is changed. With letsencrypt, this will happen more frequently - did you find a way around this?

    1. 5

      If you put nginx in front as a reverse TCP proxy, Postgres won’t need to know about TLS at all and nginx already has fancy reload capability.

      1. 3

        I was thinking about that too - and it made me also wonder whether using OpenResty along with a judicious combination of stream-lua-nginx-module and lua-resty-letsencrypt might let you do the whole thing in nginx, including automatic AOT cert updates as well as fancy reloads, without postgres needing to know anything about it at all (even if some tweaking of resty-letsencrypt might be needed).

        1. 1

          That’s funny I was just talking to someone who was having problems with “reload” not picking up certificates in nginx. Can you confirm nginx doesn’t require a restart?

          1. 1

            Hmm, I wonder if they’re not sending the SIGHUP to the right process. It does work when configured correctly.

        2. 2

          I’ve run into this issue as well with PostgreSQL deployments using an internal CA that did short lived certs.

          Does anyone know if the upstream PostgreSQL devs are aware of the issue?

          1. 19

            This is fixed in PG 10. “This allows SSL to be reconfigured without a server restart, by using pg_ctl reload, SELECT pg_reload_conf(), or sending a SIGHUP signal. However, reloading the SSL configuration does not work if the server’s SSL key requires a passphrase, as there is no way to re-prompt for the passphrase. The original configuration will apply for the life of the postmaster in that case.” from https://www.postgresql.org/docs/current/static/release-10.html

        1. 2

          If you are a fellow developer, you might find developermail.io being a fit. It’s configurable via Git and it’s supposed to be something like an “Heroku for email”.

          Disclaimer: I’m one of the creators.

          1. 2

            Seems like an interesting project.

            I would probably reach for wget (can read an a file of urls to fetch) or a combination of curl and xargs (or gnu parallel), before trying a bespoke tool like this though. That said, the X-Cache-Status statistics are neat, if you need that.

            1. 2

              That’s what I thought. When looping through a file with a few hundred thousand entries with bash/ curl, I had a throughput of ~16 requests/second, while cache_warmer easily was able to do >500 req/s.

              Thanks for the hint, I should probably add that to the post.

              1. 1

                Indeed, looping via bash would be slow due to not reusing the connection. With a carefully crafted xargs, you should be able to get multiple urls on the same line (eg. curl url1 url2 url3...). Then curl /should/ reuse a connection in that case. If curl had a ‘read urls from a file’ parameter, it would be quite a bit easier to script, but alas it currently does not.

            1. 5

              I’ve found some more information (not from original sources though, so treat with care):

              • HN thread of someone stepping down from TSC (with a lot of occurances of “SJW” in the comments)
              • Someone in a german forum did some research and claims the actual reasons are “thoughtless use of pronouns” and “assumtions of gender”.

              Personally, I’m kind of not sure whether it’s a good idea to give this too much attention. It feels like either side can only lose from this….

              1. 6

                The fact this is plausibly useful is a sad comment on the state of software engineering.

                1. 2

                  How would you describe the alternative desired state? That insecure protocols don’t exist? That engineers would have deeper knowledge of cryptography?

                  1. 8

                    Distributions of major server software would come with good configurations out of the box, alleviating every developer from being responsible for configuring things.

                    https://caddyserver.com/ is a great example of this; you configure it to do what your app needs, all the TLS defaults are well curated.

                    1. 4

                      While I agree that a “reasonably secure default” should be standard, mostly you have to find a trade-off between security and compatibility. If you need support for IE8, there’s no way around SHA. If you want to support Windows XP or Android 2, there’s no hope at all. If you want it more secure (as of today) you fence out most Androids (but 4.x), Javas, IEs, Mobile Phones, non-up-to-date browsers. Unfortunately, there is no one size fits all.

                      1. 3

                        On the other hand, compatibility with older software is very easy to figure out (people see an error message), whereas insecure configuration appears to work perfectly fine. I also believe developers are more likely to know that they need to support some obsolete software (modern web development doesn’t “just work” on IE8 or Android 2) than about the newest TLS configuration options.

                        1. 2

                          I think if you want that, we ought to have APIs that express things in terms of goals, instead of implementation details: ssl_ciphers +modern +ie8 maybe. Then it’s clear what needs to be changed to drop a platform, instead of it being a guessing game.

                          1. 2

                            This would be great. This is exactly what I’m trying to provide the user with the snippets in nginx.vim: Choose between ciphers-paranoid, ciphers-modern, ciphers-compat, ciphers-low cipher suites.

                  1. 4

                    First thought: didn’t I see this in the new nginx vim-syntax posted here a while ago?

                    Ah.. it’s by the same author.

                    Nice work, ch4r. It helped me get rid of a couple I had in my 2014 nginx ssl config. Hat tip.

                    1. 2

                      Good catch :)

                      sslsecure.vim is actually trying to get some of the security features from nginx.vim to work with other configuration files and source code as well.

                      Good to hear it actually helped you re-securing your config!

                    1. 2

                      FYI: I’ve added support for embedded syntax highlighting for ERB/ Jinja templates and LUA.

                      1. 3

                        I’m currently working on https://developermail.io - an email-SaaS that is configured with git - might be referred to as the “Heroku for email”.

                        I’d apprechiate if you consider adding it to your list!

                        1. 3

                          Oh cool! I love stuff like this - will bear in mind in case the need arises.

                          Other than exposing the sieve rules over git, how do you compare to fastmail offering more-or-less the same featureset / price point?

                          Your security page claims you use SHA-512 for password storage.

                          My understanding of crypto is… not deep, but that looks like poor marketing given http://security.stackexchange.com/questions/52041/is-using-sha-512-for-storing-passwords-tolerable is one of the first things I came across trying to figure out if it was OK.

                          1. 2

                            The unique selling point would be the git configuration, including all the advantages that git brings: You know who changed what when, you can have review processes using branches/ pull-requests, rollback changes and do bulk changes more easily. Plus you can comment your configuration. Plus: It feels more natural and leet for developers :)

                            Additionally, as you pointed out, the sieve control is way more powerful.

                            1. [Comment removed by author]

                              1. 2

                                That’s a good thought. developermail uses rounds=100000, so this should probably be mentioned on the site.

                                1. [Comment removed by author]

                                  1. 1

                                    From what I know, the used number of rounds can be treated similar to the salt. It’s not required to be hidden, as it’s stored alongside (or at least accessible with the same credentials/ from the same applications) in the system (in fact: CRYPT-SHA512 stores it in the form CRYPT-SHA512$6$rounds=xxx$HASH. In case of a data-breach security shouldn’t rely on the salt or whether the number of rounds are known.

                                    While I don’t agree with “security by obscurity”, I don’t want to escalate the discussion here.

                                    Thanks for your feedback!

                            2. 2

                              Regardig your observation for SHA-512:

                              Obviously, one would want to have script or bcrypt, maybe even Argon2 instead of SHA-512. There’s however another factor when choosing the right algorithm in my opinion, and that’s the implementation.

                              Let’s take dovecot as an example. According to the documentation SHA-512 is the strongest scheme implemented on all platforms (it mentions bcrypt but with the annotation that it’s not available on most Linux distributions). Furthermore, a Argon2/ scrypt plugin is mentioned, but it’s third party. Of course I’ve considered using one of the mentioned algorithms, do not feel competent enough to review a 3rd party plugin on my own regarding its implementation - Especially since dovecot itself was recently reviewed and received an extremly positive rating. A bad implementation of a secure algorithm may introduce other attack vectors or security issues. In case I missed something, I’d apprechiate feedback. And of course I’ll follow the ongoing implementation and new security features closely to improve the security whereever possible. I’m also wondering how other email providers handle the issue. Most of them are pretty silent on what algorithms they’re using from what I’ve observed. As anyone some insights to share?

                              TL;DR: I’d love to use scrypt() but I’m not sure whether to trust the inofficial plugins implementing it.

                              Edit: developermail.io uses rounds=100000. While one would still prefer scrypt(), this should increase computational requirements a lot. I’m going to add this to the website.

                          1. 8

                            For anyone considering this, there’s also a more elaborate fuzzy searcher called fzf that comes with shell history search shortcut

                            1. 1

                              I’m using fzf and I was wondering whether there’s any advantage using hstr (performance, packages, etc) - as it offers the same feature (and a lot more).

                            1. 4

                              The certificate for https://download.servo.org/ seems to have expired. Unfortunately, I can’t find checksums for the downloaded files elsewhere to verify the download :(

                              Super exited to try out Servo, though!

                              1. 1

                                There’s an issue already on Github

                              1. 1
                                • Safety and security. I’m looking forward to Firefox with more and more Rust components.
                                • C compatible. It’s possible to exchange more and more of your critical system components with Rust code. You can even write Kernel modules in Rust! This would eradicate so many attack vectors.
                                • Plus: The community is incredibly friendly and helpful
                                • And finally: It’s a Mozilla project. Mozilla is one of the few defenders and advocates of internet users left out there.
                                1. 8

                                  Okay, that’s great and all.

                                  What have you, chr4, personally used Rust for?

                                  You didn’t answer any of the above questions. Concrete examples please.

                                  1. 6

                                    Well, I answered the question about the Rust features I apprechiate. But you’re right, I was missing out on the examples:

                                    I’m currently rewriting some of my C projects in Rust (with more or less success), as well as tinkering around with rewriting a Go JSON API. The latter is probably not very useful, as I think Go is the better fit for the job, but it helps me improving my Rust skills. Furthermore, I’m currently working on An Interpreter In Rust.

                                    All projects so far are not for commercial projects.

                                1. 17

                                  This post lacks addressing the fundamental reasons of not using a public cloud. While it’s beneficial for tons of usecases (distributing and computing public content, like livestreams, having access to compute power for a few hours without having to buy tons of hardware, etc), it also has downsides:

                                  Security

                                  It’s a significant attack vector to share resources with others. See the Rowhammer vulnerability, or this paper (extracting private keys on shared CPU systems, there’re more papers, this was the first I found) for example.

                                  Vendor lock-in

                                  If you rely on e.g. AWS or GCE services that can’t be migrated easily (e.g. Route53), you’ve locked yourself in, and you can only switch providers if necessary with enourmous efforts.

                                  Privacy

                                  I think it’s just irresponsible to host private customer data in locations that don’t provide basic security you can trust. Your database drive might end up in the hands of a customer or third-party service contractor, let alone issues of government access. Encryption, e.g. on GCE, which relies on keys stored on their servers only address a few of those issues.

                                  Price

                                  Running a 24/7 machine on a major cloud provider usually costs a lot.

                                  Edit: Formatting

                                  1. 7

                                    Price

                                    Running a 24/7 machine on a major cloud provider usually costs a lot.

                                    This is especially true when you’re doing mainly compute, rather than hosting. Advocates of public clouds often correctly focus on the variety of infrastructure you’d have to handle yourself if you don’t go with them: uptime, several kinds of failover, fault-tolerant storage, load scaling, etc. Properly costed, that may well narrow or eliminate the price difference. But in data analysis it’s much more common to have a situation where I don’t need all that good uptime, failover, integration with databases, CDNs, etc. Instead I just need a lot of compute throughput over a period of months, for ideally as little money as possible.

                                    For example, if you’re training deep neural networks on GPUs, which is a very common current use-case both in academia and industry, AWS for a single month of usage will charge you about what it costs to buy the machine outright. Even if you add in substantial overhead, it’s hard to make AWS come out ahead here.

                                    1. 1

                                      If you rely on e.g. AWS or GCE services that can’t be migrated easily (e.g. Route53), you’ve locked yourself in, and you can only switch providers if necessary with enourmous efforts.

                                      Sad face - I have some domains in R53. Why is it so hard to migrate?

                                      1. 3

                                        I suppose you can migrate away (with the usual pain of migrating DNS) unless you use “special features”.

                                        I should’ve made it clearer in my comment: When saying “Lock-In”, a was refering to “special” features of services. If you’re running a virtual machine with PostgreSQL on a cloud provider, you can migrate to another provider and set up your virtual machine there. But if you use managed databases (like e.g. Amazon Redshift), migrating away won’t work as easily, as Redshift is not an OpenSource product you can just run anywhere.

                                        1. 1

                                          Got it. Thanks for clarifying. Yeah some of those extensions sure do have a warm embrace, huh.