Solaris Basic Security Mode (BSM) Auditing

Hal Pomeranz, Deer Run Associates

 

When enabled, the Solaris Basic Security Module (BSM) can create an extremely detailed audit trail for all processes on the system. The level of auditing produced is at the level required by systems attempting to achieve the DoD "C2" level certification. The simplest description of BSM auditing that I've been able to come up with is to imagine running truss--the Solaris system call tracing tool--on every single process on the system and saving the resulting output to a file. BSM actually provides even more detailed information than that.

 

While this information can be incredibly useful for both system administration and forensic purposes, we're obviously talking about enormous amounts of data here. As a lower bound, consider that the small workgroup server that runs my two person consulting business generates about 120MB of BSM logs per month using the configuration discussed later in this article. A machine that actually does something, like a compute server or end-user workstation, will typically generate perhaps an order of magnitude more information than my relatively meager audit trail. It's a good thing disk space is cheap these days.

 

While Sun's BSM reference[5] is quite complete, the initial stumbling block for BSM has always been a lack of clearly defined standards or recommendations on what events are necessary or useful to audit and which just generate "noise" in the audit logs. Both DISA[1] and Sun[2] have made their recommendations, and recently part of my work with the Center for Internet Security has been to rationalize these recommendations along with other input to produce some consensus standards[3]. This article is both an introduction to BSM auditing on Solaris and a discussion of the configuration recommendations we've developed via the CIS consensus project.

BSM Basics

BSM is not enabled by default under Solaris. The administrator is required to run the "bsmconv" script to set up the initial auditing environment for the system. Sun strongly recommends that the bsmconv script only be run in single-user mode, but honestly I always do my BSM configuration in multi-user mode and have never encountered any problems. Your mileage, as always, may vary.

 

bsmconv creates a number of files in the /etc/security directory. The audit_startup script is invoked at boot time and sets a number of different audit policies for the system. The audit_control file is the primary configuration file for BSM. The audit_class and audit_event files can be used when more fine-grained control of the audit configuration is required. There are a number of other BSM-related files in /etc/security, but the four files mentioned here are the most critical for this article.

 

The audit_startup script is simply a series of auditconfig commands to initialize the system auditing policy:

 

#!/bin/sh

/usr/sbin/auditconfig -conf

/usr/sbin/auditconfig -aconf

/usr/sbin/auditconfig -setpolicy none

/usr/sbin/auditconfig -setpolicy +cnt

/usr/sbin/auditconfig -setpolicy +argv,arge

 

The first two lines pull configuration information out of the audit_control file and set up the basic events that the system will audit. The remaining lines set other special auditing policy options. "-setpolicy none" first blanks the audit policy for the system so we start with a clean slate, then "setpolicy +cnt" tells the system to continue running even if the auditing partition on the machine fills up (high security sites are required to have the machine shut down if auditing becomes impossible,
"-setpolicy -cnt"), and "-setpolicy +argv,arge" means to track the full command line and all environment settings for any command executed on the system. Note that the "-setpolicy +argv,arge" line is not part of the default BSM configuration set up by the bsmconv script, but is part of the consensus recommendations from CIS[3].

 

The audit_control file appears deceptively simple:

 

dir:/var/audit

minfree:20

flags:lo,ad,pc,fm,fw,-fc,-fd,-fr

naflags:lo,ad,ex

 

"dir" is the directory where audit logs will be written on the system--it's a good idea to make sure this directory is only accessible by the superuser. There is no built-in facility for writing audit logs to some other system, though some sites have tried writing to an NFS mounted directory from some central file server (note that this configuration requires the client system to have root write privileges into the NFS volume, which has some significant security implications). "minfree" specifies the amount of free space--as a percentage--that must exist in the auditing partition or else the system starts complaining. So in our case, once our audit partition goes above 80% full, the auditing subsystem starts sending the administrator annoying warning messages (via the /etc/security/audit_warn script).

 

The "flags" and "naflags" lines are the really interesting part of the file. These lines define which audit events the system is actually going to pay attention to (these are the lines that the "auditconfig -conf" and "auditconfig -aconf" commands in audit_startup are looking at). The two letter codes are groups ("audit classes") of related events (system calls) defined via the audit_class and audit_events files. For example, the "fr" class deals with "file read" type events, which mostly consist of various permutations of the open() system call when a file is opened for reading, but also includes events like readlink() for dealing with symbolic link files. For any given class, you have the option of monitoring only system call failures ("-fr"), or only successful calls ("+fr"), or both ("fr").

 

The "flags" line defines the "audit vector" for normal user sessions on the machine. The "naflags" line catches all "non-attributable" events on the system--events that are not associated with a particular user's session. Usually, non-attributable events are the result of system processes and do not occur that frequently. Most of the interesting "tuning" for BSM happens on the "flags" line, and we'll cover these settings in a lot more detail in the latter part of this article.

 

Audit logs are written to binary files in your audit directory. The file naming convention used is "<start>.<end>.<hostname>", where <start> and <end> are time/date stamps in the format "YYYYMMDDhhmmss" and <hostname> is the fully-qualified hostname of the local machine. Actually, the current audit log that's actively being written is named "<start>.not_terminated.<hostname>" to distinguish it from the other audit logs in the directory.

 

The command "audit -n" signals the system audit daemon to close its current audit log file and start a new one--unless told otherwise, the audit daemon will simply continue writing to the current audit log and it will grow without bound until it reaches the file size limit for the machine or fills the partition. The CIS recommendation[3] is to put the following line in root's crontab to force audit logs to be restarted at the top of every hour:

 

0 * * * * /usr/sbin/audit -n

 

Once the new audit log has been started, the old log can be compressed and/or moved off of the local system for archival purposes. Some sites that are concerned about attackers sabotaging or removing the audit logs actually rotate their audit logs much more frequently (some every five minutes) so that they can move the audit logs to some secure repository elsewhere on the network.

 

Since the audit log files are binary data, you need a special tool to read them. The praudit command will dump an audit trail in a variety of different formats. The auditreduce program can be used to select particular audit events based on different criteria such as username, time of day, etc. However, the output of auditreduce is still in the internal binary format used in the audit log files themselves, so you must pipe the output of auditreduce into praudit to achieve any intelligible result.

 

Beyond that, you're sort of on your own when it comes to interpreting the audit output. Most sites end up "rolling their own" Perl scripts to parse and report on the auditing data. Some of the commercial IDS vendors have been working on incorporating the BSM audit trail into their host-based IDS products as a near real-time system auditing mechanism, but this is still far more art than science.

Tuning Auditing

As I mentioned earlier, the real difficulty with BSM is tuning the level of auditing on the system. The trick is to strike a balance between getting the events you need to reconstruct what's been happening on the system, while filtering out uninteresting events that add "noise" to the audit trail and consume huge amounts of disk space. One size does not fit all here. For example, the DoD requirements specified in the DISA STIG[1] reflect very high levels of auditing required for secure military sites. Most organizations don't require this level of auditing and can save enormous amounts of disk space by eliminating some of these audit events from their default audit vectors in the audit_control file.

 

Let me first try an explain what events some of the more useful audit classes cover, and then try and make some recommendations for preserving your sanity and your disk space.

 

General Events - "lo" (login) and "ad" (administrative)

 

The "lo" (login) class covers all forms of system logins as well as use of the su command. The "ad" (administrative) class covers a wide variety of administrative actions including rebooting the system, adding and deleting users, changing auditing and logging parameters, mounting and unmounting file both local and remote file systems, changing quotas, loading kernel modules, and even setting the system clock. Every configuration recommendation I've ever seen recommends tracking both successes and failures for events in these classes. The Sun recommendations[2] include some hints for customizing the "ad" class to reduce some of the noise from uninteresting events, but this is not a huge optimization.

 

Process Events - "ex" (execution) and "pc" (process control)

 

The "ex" and "pc" classes deal with process execution on the system. Actually, the two events in the "ex" class--exec() and execve() system calls--are also contained in the "pc" class, so if your audit vector includes "pc" then you don't need to worry about "ex".

 

Aside from the exec() and execve() system calls that are actually used to execute programs on the system, the "pc" class also tracks everything that the process might do during its lifetime--changing directories, calling setuid() and setgid() to change its privilege level, making chroot() calls, creating child processes with fork() and vfork(), etc. The "pc" class also tracks administrative interaction with processes on the system, like kill and nice.

 

The problem here is that "pc" tracks various system calls that aren't usually interesting. For example, you probably don't care to know every time your mail server fork()s a new child process to deal with an incoming connection. What you really care about is when new processes get started on the system--typically with a fork() followed by an exec(). So we really just want track the exec()s. Similarly, keeping track of every single chdir() call by a process is going to drive you nuts.

 

So what we need is a way to track the important events from the "pc" class but ignore the uninteresting ones. This requires actually creating a new custom class that includes just the events that we want. More on this later.

 

File Attribute Modification - The "fm" Class

 

The "fm" class tracks changes to file attributes like ownership (chown) and permissions (chmod) and even extended file ACL settings. However, "fm" also tracks file locking and updating timestamps on files. These latter events are way too frequent on normal Unix systems to be anything other than "noise" in your logs. Again we're going to need some way to customize the "fm" class so we see just the interesting stuff in our audit trail.

 

Other File Actions - "fc" (create), "fr" (read), "fw" (write), "fd" (delete)

 

The decision about whether or not to include the other "file" event classes to track creating, deleting, reading, and modifying files was probably the most contentious aspect of our auditing discussions within CIS. Sun[2] recommends avoiding these audit classes in order to reduce the size of the audit trail. On the other hand the DoD guidelines[1] require tracking at least failure for these classes (actually the specific recommendation is

"fw,-fc,-fd,-fr").

 

These classes really can generate an enormous number of audit events and consume huge amounts of disk space. Think about compiling a huge software package from source code--not only are you going to generate all of those new *.o files, but the compiler will probably be creating and deleting intermediate result files in some system temp directory. The "fr" class can really kill you during process execution because each process execution involves searching your LD_LIBRARY_PATH for shared library files. You have to walk through every directory in LD_LIBRARY_PATH for each *.so, and every time you "miss" and don't find the shared object in the early directories in your search path you end up generating a read "failure". I've often wished that the auditing system could distinguish between "file not found" (ENOENT) and "permission denied" (EACCES) so that I could audit one and not the other. Unfortunately, this is not possible at this time.

 

Ultimately, we decided to err on the side of caution in the CIS recommendations[3]. The default recommendation is to not turn on any auditing of these classes, though we document the DISA recommendations in the notes for the BSM item. This is not to say that these classes do not cover important events on the system (particularly from a forensic perspective), we just didn't want people following our recommendations to suddenly start running out of disk space due to their audit logging. If you have the disk space to burn, I might even be inclined to audit "fc,fd,fw"--the "fr" class just adds too much noise for my taste, but again your mileage may vary.

Custom Audit Classes

Audit class names are defined in the audit_class file. Here are the audit class entries for the classes we've talked about so far:

 

0x00000001:fr:file read

0x00000002:fw:file write

0x00000008:fm:file attribute modify

0x00000010:fc:file create

0x00000020:fd:file delete

0x00000080:pc:process

0x00000800:ad:administrative

0x00001000:lo:login or logout

0x40000000:ex:exec

0xffffffff:all:all classes

 

The first field of each line is a unique bit mask that's used to represent the audit class in the internals of the auditing subsystem (there are "gaps" in the numbering between the lines above because there are some other audit classes in the default audit_class file that I'm not showing you). The second field is the class code used in the "flags" and "naflags" lines in audit_control, and the third field is just a brief descriptive name that's for the use of the system administrator.

 

When creating a custom class, you need to pick a bit mask and a two-letter class code that are not currently in use by any other class. In the default audit_class file installed by bsmconv, bit masks from 0x00010000 through 0x08000000 are not used. Our CIS recommendations[3] create a custom class called "cc" with a bit mask of 0x08000000:

 

0x08000000:cc:CIS custom class

 

Once you've defined your new class in the audit_class file, you associate the two-letter class code with specific events via the audit_event file. Here's a sample line from that file:

 

7:AUE_EXEC:exec(2):pc,ex,cc

 

The first two fields are a unique code number and code name to identify the event. The third field is purely descriptive. The last field describes which audit classes this event is associated with. As noted earlier, exec() calls are monitored by both the "pc" and "ex" classes. We've added our custom "cc" class to the end of the line so it picks up the exec() events too.

 

So what events do we actually want to monitor with our custom class? Here's the awk code we use to modify the audit_event file in the CIS document[3]:

 

awk 'BEGIN { FS = ":"; OFS = ":" }
     ($4 ~ /fm/) && ! ($2 ~ /MCTL|FCNTL|FLOCK|UTIME/) { $4 = $4 ",cc" }
     ($4 ~ /p[cms]/) && 
         ! ($2 ~ /FORK|CHDIR|KILL|VTRACE|SETGROUPS|SETPGRP/) \
         { $4 = $4 ",cc" }
     { print }' audit_event >audit_event.new

 

Essentially what this code is saying is that we want all "fm" events except for file locking (mctl()/fcntl()/flock() handle file locking plus some other stuff) and timestamp updates with utime() plus all "pc" events except fork()/vfork(), chdir(), kill(), vtrace(), setgroups(), and setpgrp(). Check the manual pages if you have any questions about what this last set of system calls do. Once you've verified that the audit_event.new file looks the way you want it, make a backup copy of the original audit_event file and replace it with your new version.

 

Now that we've fully defined our custom class, we actually have to use it in the audit vector in the audit_control file. Here's the audit_control file from our CIS recommendations[3]:

 

dir:/var/audit

minfree:20

flags:lo,ad,cc

naflags:lo,ad,ex

 

After making all of these changes, the system must be rebooted for the changes to take effect.

BSM Caveats

It's important to mention a few additional items before closing this article:

 

 

 

Conclusion

The first stumbling block to BSM deployment seems to have been achieved, namely that we now have some reasonable configuration standards that a large number of people have agreed to. The next step is developing some good tools for reporting on events in the audit trail. The rumor is that there is already an effort in progress within Sun Microsystems to do just that. Keep your fingers crossed.

References

  1. DISA Unix STIG, http://csrc.nist.gov/pcig/STIGs/unix-stig-v4r4-091503.zip

 

  1. "Auditing in the Solaris 8 Operating Environment", William Osser and Alex Noordergraaf, http://www.sun.com/blueprints/browsesubject.html#security

 

  1. Center for Internet Security "Solaris Benchmark" document, http://www.cisecurity.com/bench_solaris.html

 

  1. OpenSSH patch to help with BSM auditing, http://bugzilla.mindrot.org/show_bug.cgi?id=125

 

  1. "SunSHIELD Basic Security Module Guide",
    http://docs.sun.com/db/doc/806-1789

About the Author

Hal Pomeranz (hal@deer-run.com) spent so much time haggling about audit flags during the CIS consensus process that he actually started having dreams about them. As scary as this may sound, this was actually an improvement over his previous subconscious activity.