File integrity assessment (FIA) tools like Tripwire(TM)[1], Samhain[2], AIDE[3], et al are commonly deployed in organizations to help assist forensic investigation after a security incident and as a host-based intrusion detection tool to help detect unauthorized file system changes (this also makes them useful monitoring tools for existing change control procedures, though that is not the focus of this article). The concept is simple: the administrator creates a configuration file that lists the critical system files and directories that the FIA tool should monitor, then uses the FIA tool to create a database that tracks common parameters about those files such as permissions and ownerships, file size, and MAC times, along with one or more cryptographic checksums over the file contents (typically via common hashing algorithms like MD5, SHA-1, etc). The FIA tool is then re-run periodically and the current state of the file system is compared to the values stored for the various files in the database--if there are any discrepancies, the files are flagged as having been modified and a report is generated.
The canonical problem with FIA tools, however, is protecting the database generated by the FIA tool, as well as the binary for the FIA tool itself, from unauthorized tampering by attackers who gain root access to the system. After all, if the attacker installs a rootkit and then updates the FIA database for the system to reflect the new state of the file system, then the administrator may be unaware of the attacker's changes. Similarly, the attacker could modify the FIA tool binary to either ignore or lie about the state of files installed by the attacker.
Several different strategies have been developed to try and address this problem:
Lately, however, I've been experimenting with a different approach. The concept is to store FIA databases and binaries for the various systems being monitored on a central, highly secure system on a protected network--call this the FIA management server. Periodically from cron, we run a script on the FIA management server that uses scp to copy the FIA database and binary to a given system, and then runs the FIA tool on that remote system via SSH. Any report output and/or updates to the FIA database can then be pulled back to the FIA management server via scp, and all traces of the FIA tool can then be removed from the remote system.
Thus the FIA tool binary and database are only available to the attacker at the instant the scan is being performed. Furthermore, there is no cron job that can be subverted by the attacker, or any other indication on the individual systems themselves that the site is even using FIA tools to monitor their systems. And as we'll see later, there are some additional tricks we can use to further deceive attackers who may be monitoring the system closely at the time the FIA scan is being performed.
One of the issues with this approach, however, is that the FIA tool must generally run with root privileges on the system being scanned. So we're assuming here that the FIA management server has remote root access via SSH to all of the systems on the network where the FIA tool is being used. This means that the FIA management server must be protected at all costs, since anybody with root access on the FIA management server will effectively have root privileges on the entire network.
On the FIA management server, use the ssh-keygen command to generate a public/private key pair for the root account. Since this authentication key will be used by a fully-automated cron job, you will probably want to use a null passphrase when creating the key--though note that this means that the private key will be stored unencrypted in the .ssh directory in root's homedir on the FIA management server. The ssh-agent utility could be used as an alternative to an unencrypted key file, but configuring ssh-agent for this purpose is outside the scope of this article.
Once the key pair has been generated on the FIA management server, copy the public key into ~root/.ssh/authorized_keys on all of the systems you plan to monitor. Also make sure that you have set "PermitRootLogin without-password" (or "PermitRootLogin nopwd" if you're using the commercial version of SSH) in the sshd_config file on these systems, so that root logins are allowed using public-key authentication. At this point you should verify that you can use SSH to execute commands from the FIA management server on the various remote systems without being prompted for a password or passphrase--something like "ssh <remotehost> uptime" should be sufficient for testing purposes.
The next step is to configure the FIA tool and create the initial FIA databases for all the systems you plan to monitor. On each system, create a directory that contains the FIA tool binary and an appropriate configuration file for that system. Next use the FIA tool to generate the initial FIA database for the machine, and store that database in the same directory with the FIA tool binary and configuration file. Now simply tar up the directory you just created, gzip it, and copy the resulting tgz file back to the FIA management server as <hostname>.tgz, where <hostname> is the name of remote system where the tarball was copied from.
[*** Note -- All of the Figures mentioned below can be found at the end of this document.]
Initially I wrote a simple script called "check" that would run an FIA scan on a single remote system. Note that for this particular implementation I'm using AIDE as my FIA tool because it's (a) Open Source, (b) portable across many Unix flavors, and (c) easy to get up and running quickly. But you could easily port my scripts to use any other common FIA tool.
After using my script for a short period of time, however, it became clear that I needed a way to more easily update the FIA databases in the various host tarballs, as well as make configuration file changes when necessary--this led to "update" and "modify-config" scripts. Since all of these scripts need to be configured with the same pathnames for the FIA install directory, database name, etc, I extracted all of this information into a single include file called "DECL" for easier maintenance.
The scripts are all fairly simple at this stage, but let's look at the "check" script (Figure 1) in more detail. After setting a sane PATH environment variable, the script checks its argument list. The only required argument is the name of the host to be scanned. The second argument, which is optional, is the SSH port number on the remote machine--on many of my Internet-facing systems I run SSH on an alternate port number so I don't have to deal with constant SSH scanning and brute force password attacks on port 22. A third argument may also be added which is the path to GNU tar on the remote system (it's easier to use GNU tar since it has built-in gzip functionality). The second argument defaults to port 22, and the third to /bin/tar (which is correct for most Open Source OS versions).
Next the script declares $ROOTDIR, which is the directory where the per-host tarballs live as well as the DECL file containing other pathname declarations. After defining a temporary file for collecting script output, the "check" script then reads in the additional settings from the DECL file.
Let's consider the DECL file briefly (Figure 2). This file defines path and file names that will be used on the remote system being scanned:
The default DECL file shown in Figure 2 has the appropriate settings to unpack the tarball as /var/spool/aide on the remote system, and assumes that the directory will contain the binary "aide", configuration file "aide.conf", and database "aide.db".
After reading all of these variable assignments from the DECL file, the "check" script does a quick sanity check to make sure that the tarball for the specified host exists on the FIA management server and initializes its temporary file. The main work of the script is a series of ssh and scp commands that are created using the command line arguments to the script and the declarations in the DECL file. The output of all of these commands is captured in $TEMPFILE for later reporting.
The first scp merely copies the tarball to the remote system. Then there's a fairly complicated ssh command, but all that's really happening here is (1) a cd to the appropriate install directory, (2) unpacking the tarball, (3) another cd into the directory created by unpacking the tarball, (4) running the FIA scan, and (5) tar/gzip-ing the updated database created by the scan so that it can be copied back to the FIA management server. These last two steps will probably require some tweaking if you decide to use a tool other than AIDE. AIDE supports the --update option which does a normal scan and simultaneously produces a new database that reflects the current state of the system, thus making it easy to update your FIA database once you've verified that all changes in the report are "expected" (the path where the new database gets dumped is defined in your aide.conf file, and must also match the $REM_NEWDB variable in the DECL file). However, if your FIA tool doesn't support similar functionality (or uses a different command line option than --update), you'll need to come up with another mechanism for running a scan and generating a new database and modify the script.
The next scp command in the script simply grabs the new tarball containing the updated FIA database created in the previous step and copies it back to the FIA management server and saves it to a file named <hostname>.tgz-update. Finally, the script uses ssh to remove the installation tarball and installation directory from the remote system, completely eliminating all traces of the FIA scan.
The only remaining item is to see whether or not there were any changes to the file system. If there are no changes, the AIDE output will contain a line of text that reads, "### All files match AIDE database. Looks okay!" If the script doesn't detect this line of output, then we're simply going to send the contents of $TEMPFILE to the standard output--since the we assume the "check" script is being run from cron, this should cause the script output to automatically be collected and emailed to the user the cron job is running as (typically root). Actually, inside of the "if" statement we're going to use awk to filter out some of the uninteresting command output from $TEMPFILE--specifically the output from scp that simply shows the file name being copied, as well as the warning banners emitted by the SSH daemon on the remote machine (which in my world begin "Authorized uses only..."). After that, it's just a matter of cleaning up $TEMPFILE.
The "update" and "modify-config" scripts are considerably simpler, but follow the same basic pattern as the "check" script, so there's not much point in reviewing them here. All of the scripts, along with a sample DECL file, as well as some sample aide.conf files can be found at my web site[5].
One issue is that an attacker who happens to be monitoring the system at the time the scan is being run is likely to notice the scan happening. If they run ps, they'll see the process in the process table, and having a directory on the system called /var/spool/aide is also likely to be a real tip-off.
But it turns out that there's no need to use pathnames like /var/spool/aide, or file names like aide, aide.conf, and aide.db. We can simply choose some arbitrary but innocuous names to try and obscure the scan from a wary attacker. This is another reason why having all of the path names abstracted into the DECL file is useful--we can easily change the names to suit our tastes, at least as long as we remember to update the file names in the host tarballs to reflect our new naming scheme.
Figure 3 shows a sample DECL file with a more generic set of file names. Using this DECL file, the AIDE directory on the remote system will end up being /var/spool/config, and the command line to run the AIDE scan will just be "./configure --config=./system.conf --update". Not much to tip off our worthy attacker.
Right now the scripts presented here are merely at "proof of concept" stage, although I am actively using them to monitor a few systems for myself and some of my customers. However, further work is needed in a couple of areas:
Still, I hope you find these scripts useful or at least hope that the ideas presented here will spur your own thinking about the use of FIA tools at your site. Feedback and updates that you make will be gratefully accepted at my email address below. Happy hunting!
[1] Tripwire(TM) home page - http://www.tripwire.com/products/index.cfm
[2] Samhain home page - http://la-samhna.de/samhain/
[3] AIDE home page - http://www.cs.tut.fi/~rammer/aide.html
[4] Jason Perlman, "Using FCheck", Sys Admin Magazine, Vol 13, No 12 (Dec 2004)
[5] Hal's AIDE info - http://www.deer-run.com/~hal/aide/
Hal Pomeranz (hal@deer-run.com) is an independent computer security consultant who has spent more time analyzing file systems than he cares to think about. No mugwumps were harmed in the creation of this article.
#!/bin/ksh
PATH=/usr/bin:/usr/local/bin
export PATH
# Command line args "check hostname [port [tarcmd]]
#
HOST=$1 # host to scan
PORT=${2:-22} # specify alternate port for SSH daemonREMTARCMD=${3:-/bin/tar} # path to GNU tar on remote machine# Local pathname definitions
#
ROOTDIR=/usr/local/aide # where tarballs live on local system
TEMPFILE=$ROOTDIR/.run$$ # temp file to use for output
# Pick up other settings
. $ROOTDIR/DECL
if [ ! -f $ROOTDIR/$HOST.tgz ]; then
echo "Tarball $ROOTDIR/$HOST.tgz does not exist!"
exit 255
fi
cp /dev/null $TEMPFILE
scp -P $PORT $ROOTDIR/$HOST.tgz ${HOST}:$REM_DIR/$REM_ARCH \>>$TEMPFILE 2>&1
ssh -p $PORT $HOST "(cd $REM_DIR; \
$REMTARCMD zxfp $REM_ARCH; \
cd $REM_IDIR; \
./$REM_CMD --config=./$REM_CONF --update; \
$REMTARCMD zcf $REM_ARCH $REM_NEWDB)" \
>>$TEMPFILE 2>&1
scp -P $PORT ${HOST}:$REM_DIR/$REM_IDIR/$REM_ARCH \$ROOTDIR/$HOST.tgz-update >>$TEMPFILE 2>&1
ssh -p $PORT $HOST /bin/rm -rf $REM_DIR/$REM_IDIR $REM_DIR/$REM_ARCH \
>>$TEMPFILE 2>&1
if [ ! "`grep '### All files match AIDE database' $TEMPFILE`" ]
then
awk "!/^Authorized/ && !/^$HOST.tgz:/ { print }" $TEMPFILEfi
rm $TEMPFILE
REM_DIR=/var/spool #parent inst dir
REM_ARCH=aide.tgz #remote tarball
REM_IDIR=aide #AIDE dir
REM_CMD=aide #aide binary
REM_CONF=aide.conf #aide config
REM_ADB=aide.db #aide DB
REM_NEWDB=aide.db.new #--update DB
REM_DIR=/var/spool #parent inst dir
REM_ARCH=conf.tgz #remote tarball
REM_IDIR=config #AIDE dir
REM_CMD=configure #aide binary
REM_CONF=system.conf #aide config
REM_ADB=conf.db #aide DB
REM_NEWDB=conf.db.new #--update DB