1. 39

Many of us are developers but some of us also maintain/deal with servers. I would like how the lobste.rs community works with its servers.

What is the list of default software you put into your servers? What do you normally install on your servers to monitr, debug and secure them?

I normally use/install:

  • tmux
  • mosh
  • fish shell
  • glances
  • htop
  • sysdig
  • Elasticsearch/Logstash/Kibana
  • Prometheus/node_exporter/

I use terraform and ansible for installing this and each particular piece of software needed.

  1. 15

    For the Windows admins (hello? Bueller?):

    • Sysinternals Suite
      Utterly essential. Every Windows system I admin has these.
    • Message Analyzer
      For packet capture & network diagnostics. Wireshark is also excellent. In general, I find MA has better protocol decoders for Microsoft specific stuff while Wireshark has much broader protocol support.
    • Debugging Tools for Windows
      Installed on an as needed basis as for crash dump analysis just perform the analysis on the client. There are some scenarios though where having them on the affected system is either required or just makes life easier. Included with the Windows SDK versus a discrete download, but the download optionally includes MSI redistributables.
    • Windows Performance Toolkit
      Installed as needed like the above and available via the Windows SDK with optional redistributables. If you’re diagnosing performance problems on a system, especially at the OS level, these are frequently invaluable. The actual analysis can be done on the client.

    For all of the above except Message Analyzer you really want to configure symbols. The Debugging Tools effectively need them to be useful in most scenarios, while the WPT tools and many Sysinternals tools are much more useful with them (e.g. viewing a call stack for a thread in Process Explorer).

    1. 2

      ProcessHacker instead of procexp ANYTIME.

    2. 22


      1. 10

        I love OpenBSD because of pf, OpenSMTPD, OpenNTPD, LibreSSL, etc. However I don’t know how to trace/debug a live server like in FreeBSD with Dtrace or in Linux with sysdig/lttng/perf/ftrace. On the other hand databases like Cassandra that use the JVM don’t play nicely on BSDs (https://issues.apache.org/jira/browse/CASSANDRA-8325). Another reason why I don’t use that much OpenBSD in production is because I am not experienced with it. That is why I tend to use FreeBSD for my erlang/backend servers due to the tracing capabilities, Linux for the databases and OpenBSD whenever I can as the gateway to the world.

      2. 8

        I use zsh instead of fish, but your list and mine are pretty similar to start. I generally administrate CentOS and Archlinux machines, so this list is a bit biased. I’d also install:

        • sssd, pointed to my FreeIPA instance
        • dehydrated, either pointed to Let’s Encrypt or a WIP internal clone of the ACME API backed by FreeIPA/Dogtag
        • I use nftables, the replacement for iptables for my personal servers. For work, I still use iptables – it’s known to be good, even if management is much more painful.
        • fail2ban, for every service. I even go and parse the HTTP logs for brute force attempts against web apps, weird or exploitable URLs, etc
        • I install and enable the Inline with Upstream Software repository, and install system-wide the latest git, ruby, php7 versions for CentOS 7.
        • A personal script that sends email alerts whenever configured systemd services fail.
        • Redhat Satellite client and configuration, though I’m also looking at replacing that with Katello eventually.
        • augeas and supporting libraries for programmatic access to config files. It makes configuration management so much smarter, and easier too: I don’t have to template entire files, or rely on iffy regular expressions.
        • I also make sure curl, wget and the full vim runtime are all installed. (The default CentOS minimal installation doesn’t come with any of those.)
        • I install some sort of NTP client – though it varies based on the server’s role. All pointed to my FreeIPA server again. Along with that, with a few exceptions, every server is set to UTC timezone. It just makes things so much easier.
        • I also use ansible to set this all up, and I’m spending some time integrating Lynis and OpenSCAP for sanity checks.

        If you couldn’t tell, I’m a sysadmin. ? All of this is way over the top for personal servers, but then, I have 5+ physical servers (some more coming in the mail this week) and numerous VMs running at home.

        1. 2

          I do a very similar thing at with my home systems, but I have to say I really dislike FreeIPA. It makes me extremely nervous how many damn services the thing runs for being such a fundamental part of a network. Maybe it’s just me being paranoid, but I really don’t think that my DNS, NTP, an HTTP server, and particularly SSL PKI should be on the same machine that is running my central kerberos server.

          Also I hope that those are the DoD STIG SCAPs!

          1. 1

            I’m generally Linux distro agnostic (I prefer *BSD), but have recently taken over a couple of CentOS servers - I was toying with the idea of moving them to Ubuntu (partly as I have more experience of Ubuntu), but would like your input on staying with CentOS.

            If you don’t mind :~)

            1. 2

              Sure! I have tons of opinions, and always appreciate someone listening to them. :)

              Anything in particular?

              1. 1

                server upgrade path process is it similar to the apt-get upgrade path?

                If I’m upgrading the OS and only have one server, how long would I expect to have it off line for a standard upgrade (approximations as in minutes / hours / days / panic :~))

                1. 2

                  It’s not too hard, or scary. I’m unclear exactly what you are asking about, so I’ll explain both situations - individual packages, or whole major version upgrades.

                  • Individual packages: In CentOS, it’s extremely safe. Unless you start adding a bunch of unofficial repositories, like IUS. Then it is as risky as those repositories are – yum does not care if a package comes from multiple sources, it will choose the version you ask for, defaulting to latest. That’s why in IUS, to install PHP 7 you need to install the package called “php7u”, not “php” as that will install the version from CentOS’s repositories. The different name is on purpose, to avoid this sort of thing happening. This is all the same as adding other repositories in Debian derivatives, and of course you will want to make sure that GPG signing keys are installed, etc etc. Katello/Spacewalk is useful as a “manual proxy”, letting you consolidate watching a bunch of repositories together and controlling when clients will see updated packages.

                  • Major version upgrades: CentOS is much more like Debian than Ubuntu. Fedora is more similar to Ubuntu in this comparison. Each version of CentOS is supported for quite a while, like Debian stable is. So how scary, to you, is upgrading from Debian 7 “wheezy” to Debian 8 “jessie”? Or perhaps, from one LTS Ubuntu release from another? 14.04 to 16.04? Now, the most recent major release, CentOS 7, is actually riskier than ones before, as it replaces the init system bringing systemd. So if you’re running CentOS 6 or older, my personal preference would be to build up a new machine alongside and switch over, rather than doing an in-place upgrade.

                  For both scenarios though, upgrades are comparatively safe. I am confident in running yum upgrade on any CentOS machine, just as I would be running apt-get upgrade on any Debian box. However, I’d more concerned upgrading an Ubuntu machine. But then, I love to run on the bleeding edge, so I use Arch a bunch too. :)

                  So just doing a run-of-the-mill server upgrade? Minutes, really. The mirror network is fast and plentiful, with PoP all over the globe. Maybe a few hours if you’re being careful, or applications need a particular order to things. Of course in all cases, backups, backups, and more backups. And some automation!

                  1. 1

                    Thanks - that’s what I should have asked - I was mainly concerned with the OS upgrade side.

                    Ah it’s good to know that CentOS 7 has the jump to systemd - I’m currently running 6.9 and planning to make the jump in the summer (if we get one in Glasgow :~)).

                    Cheers for the insight.

          2. 4

            Recently, there was this article: https://lobste.rs/s/jxwtln/3_tools_i_use_on_every_server

            Where I commented that basically, I prefer installing stuff that will just help me to manage the server. Changing shell, installing tmux or even htop should be useless. You should be able to see logs from ELK/Graylog/…, metrics from prometheus/…, and should be able to deploy/run tests/… from a repeatable way (ansible/chef/…). This is the same for Debian/OpenBSD/Windows/…

            But I understand the need of working ON a server where installing your environment is important.

            1. 4

              OpenBSD where I can.

              Linux systems get the OpenBSD treatment with OpenNTP, OpenSMTP, and daily/weekly/monthly cronjobs.

              All systems get a home grown log parser based on Artificial Ignorance and his deeper Logging notes (pdf).

              Then I mostly stay off of them.

              1. [Comment from banned user removed]

                1. 3

                  Have you ever tried pgrep? What do you think of it?

                  1. 3

                    One annoying feature of fail2ban is its ability to automatically send abuse emails. I get these emails all the time because of my Tor exits, and because of their automated nature they have no way of telling that the server is a Tor node and therefore I’m not the right person to contact.

                    The emails will accomplish nothing, if you use fail2ban please make sure you’re not sending them.

                    1. [Comment from banned user removed]

                      1. 3

                        Actually without a proper SPF and/or DKIM signature there is a 99.999% that all of those emails are marked as spam anyway and ends up never actually getting to the admin. I think you greatly overestimate how useful abuse emails are, I think I’ve gotten 1 response ever. Also tor isn’t generating the emails, the automatic abuse emails from the users are, “don’t use it” isn’t particularly useful advice.

                        1. [Comment from banned user removed]

                          1. 3

                            You simply assume that I don’t know what TOR is and that I didn’t run a TOR exit node for 5 years. I don’t really believe for a second that the vast majority of users configure their fail2ban servers to use a DKIM proxy or put SPF records for any fail2ban server. Of all the abuse emails I have gotten not a single one in my trash has a DKIM signature. I also have never gotten a response back from any of the senders.

                            1. 3

                              Your ‘good deed’ emails mostly fall on deaf ears, you’re wasting my time and yours.

                              It also helps hackers build botnets

                              We’ve heard this a million times. Everything I’d say has already been said here.

                              My point was if you’re going to operate a TOR node then don’t whine about getting emails telling you your TOR node is being used for illegal purposes.

                              The thing with abuse mails is that they take time to respond to. Sometimes, a lot of time. I am fine with this, if the senders of the abusemail have taken the time to find my email address and send it to me. If the emails are automated, however, they often go to the wrong person (a human would see the rDNS of my node, and realise that sending abuse emails to me is a waste of time) and can be sent far faster than an admin can deal with them.

                              According to many laws you could actually be held liable for any attacks launched from your open proxy.

                              As far as I’m aware, my country has no such laws. It would be an absurd idea, and make running any ISP illegal.

                              And a special niche has been carved out just for TOR node operators,

                              I don’t think this is true at all. Many ISPs do not want Tor, and others tolerate it as long as bills are paid and emails answered. I have only ever encountered one ISP who don’t even bother forwarding abuse emails to me, and I don’t count that as ‘special treatment’ or anything because, simply put, there is nothing for me to do about the abuse emails. Whoever’s sending them is either unaware of the Tor network (i.e, unqualified to send abuse emails), or a robot.

                              If it weren’t saving people’s lives from murderous dictators I guarantee node operators would be treated far more harshly

                              So you’re saying that if my completely legal server isn’t used by people suffering under oppressive reigimes, you don’t believe in my right to keep it online, relatively harassment-free? That’s not a nice precedent to set…

                              1. [Comment from banned user removed]

                                1. 4

                                  The world does not revolve around you.

                                  You’re fishing for trouble here. I think my words were clear, and I don’t appreciate you trying to apply a different intention to them.

                                  I just poked all my SysAdmin friends on IRC and they all agree that they have never gotten a legitimate abuse email that wasn’t DMCA. Again, this is uselessly aggressive, doesn’t help the conversation, and was not entirely clear.

                                  What country is that?

                                  The EFF has a great bit of documentation on this exact topic.

                                  1. 3

                                    The world does not revolve around you.

                                    I never suggested it did!! What I’m saying is that your emails are really not accomplishing anything except wasting peoples’ time. Yours, because you set them up, my ISP because they send the emails on to me, and mine, because I answer them.

                                    TOUGH! As an administrator of a network on the internet YOU ARE RESPONSIBLE FOR ALL TRAFFIC THAT ORIGINATES FROM YOUR NETWORK. Do your job and stop whining about it.

                                    If everyone thought this way, I’d get so many abuse emails that it would be totally impractical. You’d shut down half the nodes in the Tor network you claim to love overnight.

                                    What country is that?

                                    UK, well, Scotland really.

                                    Yeah I bet.

                                    I explained fully what I meant in the post. Please read it.

                                    You’re fishing for trouble here. I think my words were clear, and I don’t appreciate you trying to apply a different intention to them.

                                    No, I don’t think your words were clear at all. Can you elaborate please? You seem to be suggesting that reasonable and humane treatment is subject to circumstance.

                                    1. [Comment from banned user removed]

                                      1. 2

                                        On the contrary, many of the networks that are notified do take action.

                                        Can you show anything to support this claim? Comments made by poptart et al seem to suggest otherwise.

                                        You should probably double check the laws.

                                        Ugh, well, the legal situation doesn’t seem totally clear, but there is nothing to suggest that I should be liable, and past court rulings seem to support this. If you can find a British or Scottish law I’m violating, I’ll shut down my nodes and your poor servers can finally have peace :-)

                                        1. 2

                                          In the US there is no legal precedent set, which means that there is the risk of being made an example of, but it also means that it isn’t as black and white as simba makes it seem. Many Universities across the world run exit nodes and I’m fairly certain that they know a bit better than us, it would be interesting to have one of them chime with their thoughts.

                                          1. [Comment from banned user removed]

                                            1. 3

                                              I don’t have documents to show it, but there’s lots of people who have reported their internet access being disabled over the years because their systems are infected by malware.

                                              That’s mostly ISP’s shutting down clients that are part of botnets and are done by seeing certain types of traffic. There might be another router operator involved to pin down where the traffic is coming from, but these are not blocked by people reporting abuse. Especially in this case most of the time they are DDoS bots doing amplification attacks, no amount of abuse emails will help since they are almost always UDP IP spoofing.

                                              there is plenty of legal precident, just not at the individual scale. Data centers get threatened a few times a year about sites like Pirate Bay operating on their IPs and the result is they usually cave in to the demands of governments and law enforcement agencies, because they know the “Wasn’t us” defense won’t work in court.

                                              I worked at a University for 2 years as the only security technician, I heard this exact argument 5 times. Legal precedent is an actual legal term for a case that establishes a principle, give me the evidence in this case, because every EFF staffer (i.e. lawyer) I have ever talked to says there is none.

                                              I don’t have “proof” because my friends tend to be pretty savvy so none of them have been infected with those kinds of things.

                                              This thinking is going to get them in trouble, bullet-proof thinking makes a person arrogant and will often end up shooting the person who thinks that in the foot. I’ve had to do forensics on fully patched systems with strict SELinux rules that had extremely complex rootkits installed that the only reason we discovered their compromise was because of the amount of traffic being sent.

                                              1. 1

                                                When all the evidence shows you as the origination IP and you don’t have any logs to prove that someone else connected to you, that’s a pretty tough defense to make.

                                                I’ve got years of email logs and communication with the ISP in which I explain that the server is used as a Tor exit, and the data available from onionoo which will show that the reported ‘exit bandwidth’ according to the Tor network matches the traffic produced by the server.

                                                The fact is you could disable your open proxy and then you’d no longer be contributing to the problem.

                                                Woah, woah, woah. Are you seriously suggesting that Tor admins should just.. stop?! Do you realise how insane that sounds? Earlier in the thread you explained how you support Tor as it protects the vulnerable from tyrants. Tor is not a ‘problem’, the problem is criminal activity on the internet. The link I posted earlier explains how Tor doesn’t really boost the abuse, and you’ve no reason to suggest that it does apart from a few isolated incidents (where the criminals could have used other networks were Tor not available), so I think it’s a bit premature to call Tor a ‘problem’!

                                                Data centers get threatened a few times a year about sites like Pirate Bay operating on their IPs and the result is they usually cave in to the demands of governments and law enforcement agencies, because they know the “Wasn’t us” defense won’t work in court.

                                                No, that’s not why they cave. They cave because they have a legal requirement to shut down servers which are hosting copyright-infringing content. Tor exits do not host any content; they just relay it, so this does not apply to them.

                                                1. [Comment from banned user removed]

                                                  1. 2

                                                    Just today I’ve received emails from 4 ISPs regarding hackers using their networks, all of them positive and stating they will contact their customers as the next step.

                                                    Yeah, well, while you’re obviously pretty convinced that these emails work, I have my doubts.

                                                    I’ve received a lot of form letter responses from TOR node operators too, most of them are friendly in tone and they simply explain what a TOR node is and then they say they can’t do anything about it.

                                                    Maybe we’ve spoken already then :-)

                                                    I don’t whine about getting those emails.

                                                    Obviously not, because you sent the initial email suggesting that you expect a reply!!

                                                    If you don’t want to get notified, don’t allow people to use your server as an anonymous proxy or relay.

                                                    I don’t object to notifications at all! I happily reply to emails I get. I do, however, dislike getting automated emails for pointless things like ssh logins.

                                    2. [Comment removed by author]

                                      1. [Comment from banned user removed]

                                        1. [Comment removed by author]

                                          1. 2

                                            I guess getting 300 emails about “crawler behavior” is not spam and my spam filter doesn’t know that it has the dictionary definition wrong.

                              2. 2

                                Yes, fail2ban is great, and I run it. That said, address the root cause, not the symptom and get passwords away from your SSH auth. Moving to certificate based authentication moves the threat down to vulnerabilities in your SSH demon or the cryptography of your certificate. Those are much, much easier threats for you to control for than ‘strength’ of passwords.

                                1. [Comment from banned user removed]

                                  1. 1

                                    If you consider fail2ban a security tool you’re doing it wrong. If your web app has vulnerabilities then fix them. This is hardly a good security strategy, because many attackers spread their attack attempts throughout botnets.

                                    fail2ban is valuable, as it can help you lock out some attackers earlier and thus save resources (because an SSH login attempt takes more computing power than an iptables drop), but you really shouldn’t rely on it for security.

                                    1. 5

                                      fail2ban is a valuable security tool by adding rate limiting and delay, both valid tools in the in “protecting my system” toolbox. Fail2ban is not the whole solution.

                              3. 2
                                • monit
                                • munin
                                • graphite
                                1. 1

                                  The last time I saw the Graphite UI, it looked like cruft from the 90s. Have they updated it lately?

                                  1. 2

                                    No, it’s still ugly, but it works. There is also an alternative with a better design, Grafana, but I haven’t used it.

                                2. 2

                                  I just have a small homeserver, but I documented it.

                                  Not to repeat things, additional recommendations from me would be:

                                  • etckeeper (keeps /etc in git implicitly)
                                  • unattended-upgrades (Debian/Ubuntu specific)
                                  • cronic (to make cron be more quiet)
                                  1. 2

                                    I use:

                                    • NixOS to have configuration files be immutable
                                    • qpress for managing file compression on backups
                                    • collectd / graphite for stats
                                    • a very simple emailer for failure problems
                                    • fail2ban on some critical ports
                                    1. 2
                                      • Monitor: collectd (will try node_exporter one of these days, but collectd works well)
                                      • Debug: strace/gdb since forever. Now perf.
                                      • Secure: Always default deny incoming connections except ssh, http, https via iptables. Install fail2ban.

                                      Other tools/changes I often make:

                                      • Disallow root login via ssh
                                      • Disallow ssh password auth
                                      • iotop/htop/atop
                                      • jq because many apps have some json lying around
                                      • tcpdump/tcpflow
                                      1. 2

                                        I’m SHOCKED nobody has mentioned nmon - http://nmon.sf.net/ - It’s from IBM originating in AIX, but is now open source and runs on various UNIX systems. It also has various tools to work with data it can export in clever ways, and a set of spreadsheets that work in conjunction with it over at https://www.ibm.com/developerworks/aix/library/au-nmon_analyser/

                                        1. 1

                                          Right now we just install the Mesos agent and run everything on container images. Some prometheus exporter images and that’s about it.

                                          1. 1

                                            A fairly recent addition to my stack is netdata – I know it looks way over the top, but it’s really easy on resources and makes debugging issues / reading up on system events so much easier.

                                            1. 1

                                              My regular checklist for a new server:

                                              • Distribution installed with a minimal list of packages, usually CentOS.
                                              • Firewall (usually iptables) installed and enabled. Default drop both inbound and outbound, whitelist ports as needed for whatever the application is.
                                              • sshd configured for no root login, no password logins. Differently-named admin user added with ssh key auth and sudo.
                                              • All logs forwarding to a central log server. (Usually an ELK infrastructure, but I can get by with a regular syslog server that just writes to files.)
                                              • Basic Linux debugging tools: tcpdump, ethtool, netstat, ss, strace, lsof, and friends. Perf and SystemTap are nice bonuses.
                                              • When possible: each externally-facing application running on the server should be running as its own user. Apps should never be root without a very good reason.
                                              • All of the above configured using a CM tool, usually Ansible. Ideally this should be a server I never have to log into. :)

                                              I generally try to avoid installing alternate shells; customized dotfiles; or anything that doesn’t exist in the distro default packages. I spend way too much time logged into servers with strict rules about software approvals, or which are just owned by other people, so I don’t want to become dependent on anything outside the system default experience.

                                              1. 1

                                                Disclaimer being that I work for sentry, but for the simple servers and scripts that I run on my VPS, i always integrate sentry on them. I find it convenient as a first resort for “why did my twitter bot break” type questions.

                                                None of the applications that I run right now are important or complex enough for me to worry about configuring system-level tools.