Our contemplation of filesystem security isn't complete without a mention of rootkits.

After a successful root compromise, attackers might upload and install a rootkit, which is a collection of replacement system programs that enable attackers to hide their tracks and easily reconnect to the system at a later time. It is not unusual for an attacker to patch the hole that enabled him to gain access, to avoid losing the system to another attacker.

Rootkits typically include replacements for the following commands:

· ps. Shows process information. The rootkit version hides processes run by the attacker—they simply don't show up in the output.

· netstat. Shows network connections, routing information, and statistics. Attackers certainly don't want you discovering them connected to your systems. So they install a modified netstat binary that effectively cloaks connections on specific ports or specific client addresses.

· ifconfig. The attacker might want to sniff the network to pick off authentication credentials (among other things). To do this, the network interface card must be put into promiscuous mode. A very observant administrator might notice the "P" flag in the output of the ifconfig command. The modified netstat doesn't print the "P".

· df. Shows filesystem free space and inode usage. The attackers'toolkit and sniffer logs consume diskspace that might be noticed on a quiet system. The rootkit df ignores files stored in a particular directory or owned by a particular user id.

· ls. Lists files. Similar to the modified df, the rootkit ls behaves just like the standard ls but does not report files contained in a hidden directory or owned by a particular user id.

· sum. Calculates checksum and block counts. Should the administrator become suspicious and attempt to checksum the files against known good files (on a "clean" system), the rootkit sum program will produce faked checksum values that match the original binaries. Never ever rely on sum for security. It is possible for an astute attacker to create modified programs that still output the same sum value as the originals. (Instead, use cryptographic routines like MD5.)

Rootkits are readily available across a range of operating systems and architectures regardless of public availability of source. Patching binaries to include rogue code is not rocket science. It involves an understanding of binary file formats (for example, ELF) and some file manipulation—so don't assume you're invincible just because you are running a closed source OS.

Rootkit Countermeasures

The primary method to detect the presence of a rootkit is to use integrity assessment software. These programs take a digital snapshot of the system and alert you to changes to important system files.

When you first install integrity assessment software, a baseline database is created. This contains a unique signature for every file that is to be watched. Then, on a basis set by the administrator, new signatures are generated and compared with those stored in the integrity database. A mismatch means a file has been modified in some way—possibly indicating your system has been compromised. Alternatively, it could just mean you've applied an OS patch!

It used to be that system administrators would use a program like sum to generate file signatures of important system files. However, as they were to learn, these signatures can be faked. Attackers were able to cash in on the weaknesses of these checksum generators (or simply replace the program), thereby fooling the administrator.

In 1992, Gene Kim and Gene Spafford developed the Tripwire tool. Tripwire made use of digital hashing algorithms such as MD5, to create file signatures that were impractical to forge. Even the slightest change to a file, or to the file's inode information, resulted in an unmistakably different hash. The software filled a real gap in the security toolkit and proved incredibly popular. It was ported to numerous platforms and became the de facto integrity assessment software, referenced in just about every security book you'll ever come across.

This software is today known as Tripwire ASR (Academic Source Release) version 1.3.1.

For all its good points, this software has a major limitation—the database must be stored on read-only media like a write-protected floppy disk or tape. Not surprisingly, this is an inconvenience and doesn't scale well. It may be the most common reason why sites give up using Tripwire. Storing the database on a read-only filesystem doesn't cut it either—an attacker can simply remount it read-write.

Realizing there was something of a market for this kind of tool, and that there was some mileage to be had in a major update, the authors set up a company, Tripwire Inc. This breathed life back into a popular tool. A number of new features were added, and the product was fully commercialized.

Possibly the most useful feature added is that you are no longer forced to store the integrity database on write-protected media—the database itself is signed using a 1,024-bit El Gamal encryption algorithm—you can store the integrity database on the system itself. That's not to say storing it on write-protected hardware is not a sensible idea. But, if that is not sustainable in your environment, then this may be for you.

Tripwire uses a policy language to define what to monitor. Check out the included documentation for a useful tutorial. The commercial version is somewhat easier to configure (no compiler required and the policy language seems friendlier) and ships with some reasonable defaults. Whatever version you use, though, don't forget to add in your application files and create the baseline database as soon after OS installation as possible.

For a stealthy way of running Tripwire, consider this. Create a cron job on a separate (hardened) system to remotely copy across the binary and database files and invoke the comparison. Don't forget to erase the files after the check. (The output can be stored on the invoking system ready for checking/filtering.)

While the commercial version does offer some worthwhile benefits, don't feel you have to pay out to get the core benefit—many sites still run the ASR version without a hitch.

The commercial version is now available for Windows NT. Those running larger sites might be interested in the Tripwire HQ Manager product to centralize management of all V2+ UNIX/NT Tripwire agents.

Good news if you're a Linux user, though—the Linux port of the commercial version was made open source and can be downloaded free.

Binary and source copies are available from http://www.tripwire.org.

The commercially supported product is available here: http://www.tripwire.com

Kernel Rootkits

We've covered userland rootkits—with which an attacker compromises the system and replaces important system files, and now we examine the ultimate form of deception—the kernel rootkit.

To appreciate the stealth provided by a kernel rootkit, it is vital to understand the role played by the kernel. The kernel is the huge C program that runs the show—it operates at a low level interfacing directly with system hardware. An attacker who reprograms the kernel can change the behavior of the system in any way he chooses. Consequently, if an attacker modifies the kernel, he can literally change your world. Unless the attacker leaves the digital equivalent of muddy feet on the carpet, you'll probably never even know about the attack.

The means to introduce a kernel rootkit is a root level compromise (just as for a standard rootkit). The usual purpose is to hide cracker activity and provide a convenient way for crackers to reconnect later on.

A kernel rootkit really is the most devious form of back door—it is the ultimate cloaking device. All bets are off when the kernel has been subverted.

Kernel rootkits typically modify the kernel call table to redirect system calls to rogue code introduced by the attacker. The rogue code performs whatever actions the cracker intends and then calls the original OS code to let the call complete. The user is kept blissfully unaware of this.

A typical kernel rootkit

· Hides processes. No matter what tool the administrator uses, the attackers'processes are hidden—the kernel lies. This overcomes the limitations of backdooring.

· Modifies system logging routines (process accounting, C2 kernel audit, utmp, and so on).

· Hides network connections.

· Modifies NIC (Network Interface Card) status to hide sniffer activity.

· Reports false file modification times.

· Hides the presence of the module (in the case of an LKM).

· Does anything else the attacker can think of.

Three main methods exist to introduce rogue code into the kernel:

· Modifying kernel memory on a live system via /dev/[kmem|mem]

· Patching the kernel binary on disk

· Loading a kernel module

Traditionally, kernels were monolithic—a big slab of code did everything. Modern UNIX systems support Loadable Kernel Modules (LKM), which enable the administrator to introduce new kernel code into the operating system whilst the system is running. This could be done to provide support for additional filesystem types, network drivers, or custom security routines. Check out your man pages for the following insertion commands: insmod, lsmod, and rmmod.

Whatever insertion method is used, kernel integrity is paramount—if the new code doesn't behave and tramples over key kernel structures, then the system is likely to crash. This isn't too subtle. Developers of kernel code know this all too well.

The act of patching a live kernel is actually less scary than it sounds (as long as you do it correctly). This technique is sometimes used to tweak kernel parameters where no "userland" utility exists. (It is generally unsupported, however.) Inserting new kernel code involves locating and overwriting unused areas of kernel space with your code and repatching the system call table to divert callers to your code.

Kernel patching on disk involves writing your changes directly to the kernel image using a binary patcher. You seek through the binary to specific locations and overwrite with your own code. File headers are likely to need modifying, so a basic understanding of object file formats is required.

All other previously mentioned methods are possible if root access has been gained.

LKMs, however, provide the most convenient method for backdooring the kernel. Consequently, LKMs appear to be the most common delivery mechanism for rogue kernel code in the wild. Inversely, LKMs provide the good guys with a way to enhance existing security, too.

At this point you might be thinking that closed source operating systems should be safe from this kind of thing. Again, as with standard rootkits, access to source is not a major factor. (Besides, source code for some closed source operating systems circulates within the underground community.) Kernel hacking requires a familiarity with kernel structures (documented in /usr/include), some skills with a kernel debugger, and an appreciation of kernel issues (for example, how to allocate memory correctly).

For example, in December 1999, mail was sent to Bugtraq announcing the availability of a Solaris Loadable Kernel Module back door. The note was from Plasmoid, a member of The Hackers Choice (THC)—a Germany-based group with some very talented individuals.

As with any program, it only takes one person to codify a kernel rootkit with a friendly userland interface. Then anyone can install and use it on a compromised system.

Even if you can't find a rootkit for your system, it is probably prudent to assume a kernel rootkit exists, and, therefore, you should implement countermeasures. This might sound like unnecessary paranoia—perhaps it is. On the other hand, bear in mind that the cracker community is very effective at sharing tools. Crackers don't tend to advertise their tools with big neon signs, though.

Protecting Against Kernel Attacks

Safeguarding against kernel attacks can be summed up in one word: prevention. You need to prevent attackers from writing to kernel memory (directly or indirectly through LKMs) or the on-disk kernel. This is easier said than done because, if they have root, they can modify the kernel. To prevent this, you need to get in there first and change the rules. But, to do this, you need to modify the kernel itself—and, if your OS doesn't have LKM support, you're on your own.

If you're wondering why Tripwire isn't mentioned as a countermeasure here, you should probably re-read the introduction to this article. Tripwire is a userland tool. It doesn't run in the kernel—it makes calls to the kernel and bases its decisions on values returned by the kernel. Knowing this, an attacker can re-route calls made by Tripwire, to custom code that generates false checksums that match Tripwire's expectations. This is usually implemented by a rootkit in order to hide the presence of the rootkit on disk.

Rootkit Detection

If you can't prevent, then detect. This is a sound security principle that can be applied to many security-related situations.

Currently, there is no generalized method to identify whether a given kernel on an arbitrary system has been subject to a rootkit attack. However, there are specific detect points for a number of published back doors.

Authors of kernel rootkits might include a routine to identify that the kernel rootkit is actually inserted. For example, the Adore LKM back door written by Stealth can be detected by making a call to setuid with a magic number. If you supply the right number, the kernel module announces its presence. Of course, you are relying on defaults here. If your attackers used any sense at all, they would have modified the magic number or even the particular call used, and this crude detection scheme would fail.

A program that implements a number of checks for common back door modules on Linux is rkscan. Using the technique outlined previously, rkscan can identify multiple versions of popular rootkits, Adore and Knark (written by Creed). Rkscan is available from http://www.hsc.fr/ressources/outils/rkscan/index.html.en.