Detecting Trojans is easy, if you have a static search pattern to scan for. Anti-virus software routinely detects (some) Trojans using much the same pattern-searching techniques used for detecting viruses. However, identification of a known Trojan isn't always the best defense. Detection of previously unknown Trojans is also (conceptually) simple, provided you have always maintained the best security practices (literally always, at least as far as the protected system is concerned).

Most detection methods on traditional multiuser systems derive from a principle sometimes called object reconciliation. Object reconciliation is a fancy way of asking, "Are things still just the way I left them?" Here's how it works: Objects are system areas, such as files or directories. Reconciliation is the process of comparing those objects against a snapshot record of the same objects taken at some previous date when the protected object was known to be in a trustworthy, "clean" state.

Note

Strictly speaking, there is no such "clean state" time. Even a "day zero" installation of system software onto a virgin system assumes a program suite with no substitutions and back doors. Can you place unlimited trust in a system you didn't build totally from scratch yourself? "No amount of source-level verification or scrutiny will protect you from using untrusted code," says Ken Thompson in his Reflections on Trusting Trust. Does this mean we should give up on this approach? Of course not, but we should bear in mind that, even if we build an applica tion from scrutinized source code, we might not be able to trust the compiler or every snippet of hardware microcode on the system motherboard.

More commonly, the process described as object reconciliation is known as change detection, integrity checking, or integrity management. However, these terms are not strictly synonymous.

Change detection simply describes any technique that alerts the user to the fact that an object has been changed in some respect.

Integrity checking has the same core meaning, but is often taken to imply a more sophisticated approach, not only to detecting change in spite of attempts to conceal it, but to ensuring that the reporting software itself is not subverted.

Integrity management is a more general term. It can include not only the detection of unauthorized changes, but other methods of maintaining system integrity. Such methods can include some or all of the following, in no particular order:

• Maintaining trusted backups
• Blocking unknown intrusions at entry (for instance, by running system files from read-only media or refreshing system files from trusted read-only media)
• Maintenance of strict access control
• Careful application of manufacturer patches to block newly discovered loopholes
• A finely engineered change-management system, using only signed (trusted) code.

A simple method of testing file integrity, is based on reports of changes in file state information. Different file integrity tests vary in sophistication. For example, you can crudely test a file's integrity using any of the following indexes:

• Date last modified
• File creation date
• File size

Unfortunately, none of these three methods constitute a really adequate defense against more than the crudest attack. Each time a file is altered, its values change. For example, each time the file is opened, altered, and saved, a new last-modified date emerges. However, this date can be easily manipulated. Consider manipulating this file timestamp. How difficult is it? Change the system time, apply the desired edits, archive the file, and re-set the system time. Better still, get and save the date/time information using standard C library functions (for instance), modify or replace the object, and restore the file modification date. On a single-user system (like MS-DOS) with minimal or no access controls, the coding involved is trivial. For this rea son, checking time of modification is an unreliable way to detect change. Also, the last date of modification reveals nothing if the file was unaltered (for example, if it was only copied, viewed, or mailed). On the other hand, if there is a disparity between the modification date returned by the system and the date of modification recorded by a system monitoring utility, there is a distinct possibility of malicious action.

Another way to check the integrity of a file is by examining its size. However, this value can be very easily manipulated, either by trimming or padding the file itself, or by altering the value reported by the operating system.

There are other indexes. For example, basic checksums could be used. However, although checksums are more reliable than time and date stamping, they can be altered, too. If you rely on a basic checksum system (or use change detection software, which relies on simple checksumming), it is particularly important that you keep your checksum list in a trusted environment. This might mean on a separate server or even a separate medium, accessible only by root or other trusted users. Checksums work efficiently and appropriately for checking the integrity of a file transferred, for example, from point A to point B, but are not suitable for high security applications. They simply aren't designed to guard against a malicious attempt to subvert them to return false information.

A less easily subverted technique involves calculating a more sophisticated digital fingerprint for each file using various algorithms. A family of algorithms called the MD series can be used for this purpose. One of the most popular implementations is a system called MD5.

MD5

MD5 belongs to a family of one-way hash functions called message digest algorithms. The MD5 system is defined in RFC 1321 as follows:

The algorithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given prespecified target message digest. The MD5 algorithm is intended for digital signature applications, where a large file must be "compressed" in a secure manner before being encrypted with a private (secret) key under a public-key cryptosystem such as RSA.

When you run a file through MD5, the fingerprint emerges as a 32-character value. It looks like this:

2d50b2bffb537cc4e637dd1f07a187f4

Many sites that distribute UNIX software use MD5 to generate digital fingerprints for their distributions. As you browse their directories, you can examine the original digital fingerprint of each file. A typical directory listing might look like this:

MD5 (wn-1.17.8.tar.gz) = 2f52aadd1defeda5bad91da8efc0f980

MD5 (wn-1.17.7.tar.gz) = b92916d83f377b143360f068df6d8116

MD5 (wn-1.17.6.tar.gz) = 18d02b9f24a49dee239a78ecfaf9c6fa

MD5 (wn-1.17.5.tar.gz) = 0cf8f8d0145bb7678abcc518f0cb39e9

MD5 (wn-1.17.4.tar.gz) = 4afe7c522ebe0377269da0c7f26ef6b8

MD5 (wn-1.17.3.tar.gz) = aaf3c2b1c4eaa3ebb37e8227e3327856

MD5 (wn-1.17.2.tar.gz) = 9b29eaa366d4f4dc6de6489e1e844fb9

MD5 (wn-1.17.1.tar.gz) = 91759da54792f1cab743a034542107d0

MD5 (wn-1.17.0.tar.gz) = 32f6eb7f69b4bdc64a163bf744923b41

If you download a file from such a server and find that the digital fingerprint of the downloaded file is different, there is a good chance that something is amiss.

With or without MD5, integrity management is a complex process. Various utilities have been designed to assist with integrity management on complex and distributed systems. The following utilities were originally UNIX-based, but similar programs are available for Microsoft operating systems.

Tripwire

Tripwire (written in 1992) is a comprehensive file integrity tool. Tripwire is well designed, easily understood, and easily implemented.

The original values (digital fingerprints) for files to be monitored are kept within a database file. That database file in simple ASCII format is accessed whenever a signature needs to be calculated or verified.

Ideally, a tool such as Tripwire would be used immediately after a fresh (day zero) installation. This gives you 100% assurance of file system integrity as a starting point (or nearly 100%—remember the Ken Thompson article). Once you generate the complete database for your file system, you can introduce other users (who will immediately fill your system with junk that, optionally, may also be fingerprinted and verified on subsequent checks). Here are some of its more useful features:

• Tripwire can perform its task over network connections. Therefore, it's possible to generate a database of digital fingerprints for some entire networks at installation time.
• Tripwire was written in C with a mind toward portability. It will compile for many platforms without alteration.
• Tripwire comes with a macro-processing language, so that your tasks can be automated.

Tripwire is a popular and effective tool, but there are some security issues common to most or all integrity management tools. One such issue relates to the database of values that is generated and maintained. From the beginning, Tripwire's authors were well aware of this:

The database used by the integrity checker should be protected from unauthorized modifications; an intruder who can change the database can subvert the entire integrity checking scheme.

One method of protecting the database is to store it on read-only media. This eliminates any likelihood of tampering. Kim and Spafford suggest that the database be protected in this manner, although they point out that this could present some practical, procedural problems. Much depends upon how often the database will be updated, and its size. Certainly, if you are implementing Tripwire or a similar utility on a wide scale (and using its most stringent settings), the maintenance of a read-only database could be formidable. As usual, this breaks down to a trade-off between the level of risk and the inconvenience of setting and maintaining paranoid defaults.

TAMU

The TAMU Tiger suite (from Texas A&M University) is a collection of tools that greatly en hance the security of a UNIX box. These tools were created in-house, in response to an extensive attack from a coordinated group of Internet crackers. The package has been upgraded and renamed TARA (Tiger Analytical Research Assistant). It incorporates a number of scripts used to scan UNIX systems for problems.

Hobgoblin

Hobgoblin is an interesting implementation of file- and system-integrity checking. It is both a language and an interpreter. The language, according to the authors, describes the properties of a set of files, and the interpreter checks whether the description matches the actual files, and flags any exceptions.

On Other Platforms

File integrity checkers exist for Windows, (in fact there is an implementation of Tripwire for Windows NT). Integrity checkers are not necessarily expressly designed to check multiple machines and file systems over networks. Some older DOS and Windows tools use simple CRC checksumming as an index and therefore might be easier to subvert than tools that employ MD5 and related algorithms. The majority are intended for use as a supplement to virus scanners (since detectable changes to an infectable object might indicate virus infection). This doesn't invalidate the potential usefulness of integrity checkers as a means of detecting possible substitutions of compromised code for system files.

However, change detection is less convenient on Windows platforms in that system files accessed by multiple applications can be replaced by legitimate installations and upgrades. There is often a sharper delineation on other platforms between files belonging to the system and files that belong to an application.

Furthermore, change detection only works well with certain types of binary executables, even in the context of virus detection. Many viruses and Trojans infect files whose main purpose is to contain data (spreadsheets, word-processing files, and so on). However, such files are usually intended to be modified, as are the log files used on many multiuser systems to track possible malicious action. Clearly, change detection based on the presumption that files remain static isn't going to work in these instances. In some instances, it's possible to specify changes that might signify a breach (the addition of macro code to a Word file, for instance). This approach requires that the inspecting software "know" more about the internals of the file, rather than just its digital fingerprint. That would entail serious administrative difficulties, so the approach is not well favored at present.

The safest defense, though, is to block unauthorized modification of system files proactively by code signing, read-only media, and other pre-emptive measures.