The security audits triggered by Heartbleed lead to an increasing discovery of security issues in libraries such as GnuTLS and OpenSSL within the past weeks. These issues were dealt with responsibly, i.e. they were usually published together with the corresponding binary updates by major Linux distributions such as Debian (subscribing to debian-security-announce or comparable sources is highly recommended if you want to keep track of these developments).

After downloading an updated binary of a shared library, it is important to restart all services (processes) that are linked against this library in order to make the update take effect. Usually, processes load a shared library code segment to random access memory once during startup (actually, the program loader / runtime linker does this), and do not reload that code afterwards throughout their life time, which may be weeks or months in case of server processes. Prime examples are Nginx/Apache/Exim being linked against OpenSSL or GnuTLS: if you update the latter but do not restart the former, you have changed your disk contents but not updated your service. In the worst case the system is still vulnerable. It should therefore be the habit of a good system administrator to question which services are using a certain shared library and to restart them after a shared library update, if required.

There is a neat helper utility in Debian, called checkrestart. It comes with the debian-goodies package. First of all, what is debian-goodies? Let’s see (quote from Wheezy’s package description):

Small toolbox-style utilities for Debian systems
 
These programs are designed to integrate with standard shell tools, extending them to operate on the Debian packaging system.
 
 dgrep  - Search all files in specified packages for a regex
 dglob  - Generate a list of package names which match a pattern
 
These are also included, because they are useful and don't justify their own packages:
 
 debget          - Fetch a .deb for a package in APT's database
 dpigs           - Show which installed packages occupy the most space
 debman          - Easily view man pages from a binary .deb without extracting
 debmany         - Select manpages of installed or uninstalled packages
 checkrestart    - Help to find and restart processes which are using old
                   versions of upgraded files (such as libraries)
 popbugs         - Display a customized release-critical bug list based on
                   packages you use (using popularity-contest data)
 which-pkg-broke - find which package might have broken another

Checkrestart is a Python application wrapping lsof (“list open files”). It tries to identify files used by processes that are not in the file system anymore. How so?

Note that during an update a certain binary file becomes replaced: the new version is first downloaded to disk and then rename()ed in order to overwrite the original. During POSIX rename() the old file becomes deleted. But the old file is still in use! The standard says that if any process still has a file open during its deletion, that file will remain “in existence” until the last file descriptor referring to it is closed. While these files that are still held “in existence” for running processes by the operating system, they are not listed in the file system anymore. They can however easily be identified via the lsof tool. And this is exactly what checkrestart does.

Hence, checkrestart “compares” the open files used by running processes to the corresponding files in the file system. If the file system contains other (e.g. newer) data than the process is currently using, then checkrestart proposes to restart that process. In a tidy server environment, this usually is the case only for updated shared library files. Below you can find example output after updating Java, Python, OpenSSL, and GnuTLS:

# checkrestart
Found 12 processes using old versions of upgraded files
(5 distinct programs)
(5 distinct packages)
 
Of these, 3 seem to contain init scripts which can be used to restart them:
The following packages seem to have init scripts that could be used
to restart them:
nginx-extras:
    20534   /usr/sbin/nginx
    20533   /usr/sbin/nginx
    20532   /usr/sbin/nginx
    19113   /usr/sbin/nginx
openssh-server:
    3124    /usr/sbin/sshd
    22964   /usr/sbin/sshd
    25724   /usr/sbin/sshd
    22953   /usr/sbin/sshd
    25719   /usr/sbin/sshd
exim4-daemon-light:
    3538    /usr/sbin/exim4
 
These are the init scripts:
service nginx restart
service ssh restart
service exim4 restart
 
These processes do not seem to have an associated init script to restart them:
python2.7-minimal:
    2548    /usr/bin/python2.7
openjdk-7-jre-headless:amd64:
    4348    /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java

Nginx (web server) and OpenSSH (SSH server) are linked against OpenSSL, and Exim (mail transfer agent) is linked against GnuTLS. So that output makes sense. Obviously, after an update of Python and Java processes using this interpreter or VM (respectively) also need to be restarted in order to use the new code. Checkrestart is extremely helpful, thanks for this nice tool.

Update 2014-09-03:

With the upcoming Debian 8 (Jessie, currently unstable), a new package has been introduced — needrestart. It is a more modern version of checkrestart, and tightly integrates with the systemd service infrastructure. Notably, needrestart comes in its own package and with the goal to become more popular than checkrestart (which was hidden in debian-goodies, as discussed above). Currently, a discussion is going on on the debian-security mailing list about installing and running needrestart in a default Debian installation.