IPSec was designed by a dedicated working group of the Internet Engineering Task Force (IETF). The goal behind IPSec creation was the development of a single standard providing high-quality, interoperable, and flexible security for both IPv4 and IPv6 networks. The development was initiated from the needs of an Automotive Network Exchange (ANX) that required a safe interconnection among multiple vendors, suppliers, and customers.

The IP Security Protocol Working Group develops mechanisms for protection of IP traffic through defining the structure of the protected IP packets and implementing the security associations used for VPN communications. Even though the protocol itself is not finalized concerning its key management issues, it does define specific protocols for data authentication, confidentiality, and integrity.

We have already mentioned that the IPSec protocol consists of three main parts that define two (AH and ESP) modes of its operation:

• AH provides data origin authentication, connectionless integrity, and an antireplay service.

• ESP provides data origin authentication, connectionless integrity, antireplay service, data confidentiality, and limited traffic flow confidentiality.

• IKE provides cryptographic algorithm negotiation and distributes the keys utilized by both AH and ESP.

Security Associations

Both AH and ESP rely on the security associations (SAs) negotiating the properties of a secure connection using IKE. SA holds information negotiated between two VPN participants. Such information includes cryptographic keys and their lifetimes, cryptographic algorithms used, IPSec protocol, and its mode of operation.

For each mode of operation, two SAs are required, one for incoming and one for outgoing traffic. Two SAs describing the data destined for and originating from the host are called an SA bundle. If both modes of operation (AH and ESP) are used, it would require the negotiation of four SAs. Each SA is specifically identified by AH or ESP protocol, destination IP for an outgoing or source IP for an incoming connection, and a 32-bit integer (SPI) used as a unique identifier. Another important feature of each SA is its lifetime. The lifetime parameter specifies the time interval after which the SA must be renegotiated or terminated. The lifetime is specified as a number of bytes processed or as a time interval; whichever criteria is reached first, the SA is renegotiated. Apparently, two different limits exist for the SA lifetime: hard and soft. When the soft limit is reached, the SA is renegotiated, but it is not until the hard limit is reached that the old SA is terminated from the host's memory.

Each host participating in the communication stores SA information in its Security Association Database (SAD). In fact, a second database is necessary for proper IPSec operation, the Security Policy Database (SPD), which holds the information on policies to be applied to the traffic. SPD consists of a ruleset, further split into a number of selectors that carry information on the type of action to be performed. Once the packet arrives, it is checked against the SPD database for a high-level decision on what to do with the packet next, whether the packet should be discarded, passed on, or subject to processing by IPSec. In contrast to the SPD, SAD supplies the necessary parameters for the connection.

To decide what to do with a packet, three fields are extracted from the packet header and matched against the respective SAD (IPSec protocol, IP address, and SPI). If a match is found, the parameters are further matched against the fields in AH or ESP. If no match is found, the packet is discarded.

AH

AH is one of the protocols within IPSec that allows you to check the authenticity of the data and header of the IP packet. It does not provide a mechanism for data encryption, but it provides a hash that allows you to check whether the packet was tampered with along the way. This form of encapsulation alone has gained rather limited use, as more people tend to use ESP alone or a combination of ESP and AH.

AH also accounts for reply attack protection by using sequence numbers in the packets that it sends out and implementing a sliding window on each IPSec node. Once the IP packet is received, the sliding window is advanced, so that any packets that arrive outside this window are dropped. The same applies to the packets with sequence numbers that are repeated.

The authentication of a packet is provided using HMACs. If any part of the IP header field or data field has been modified, the HMAC message digest calculated at the receiving host would differ from the original hash, meaning that the packet was modified in transit. Thus, the integrity of transmitted data is checked.

IPSec uses HMACs employing various one-way hash functions such as MD5, SHA-1/2, and RIPEMD-160.

AH can operate in tunnel and transport modes and is classified in RFC 2402 as the protocol type 51.

ESP

ESP provides for encapsulation of the unprotected IP packet, its encryption, and its authentication

Traditionally, IPSec uses DES or 3DES encryption. DES is considered to be weak and can be broken in a matter of days or even hours if needed, so its use is not recommended. 3DES, sometimes referred to as "DES on steroids," provides much stronger encryption, but the algorithm is mathematically intensive and pretty slow on devices with limited processing power, such as access points, older PCs, or handhelds. For data integrity protection, MD5 or SHA-1 are commonly used to calculate hashes on the data included in a packet. Replay attack detection works in a similar manner to replay detection in AH. ESP is classified as protocol 50 and is defined in RFC 2402.

Alternative IPSec implementations exist that use much stronger encryption ciphers, such as Rijndael and other final-round AES candidates. More secure cryptographic hash functions like SHA-2 and RIPEMD are also supported.

IP Compression

The addition of extra headers to the IP packet after encapsulation results in an increase in packet size, creating tunnel overhead. The addition can be as much as 300 bytes for ESP encapsulated traffic. If AH is used in conjunction with ESP, the resulting overhead is increased even further. This negatively affects the performance of the communication, as the real throughput of the network decreases. As compared to modern wired LANs, wireless networks have lower bandwidth and throughput, making additional overhead highly undesirable.

IPSec tries to combat this problem with a built-in IP compression (IPComp) protocol that generally utilizes the DEFLATE or LZS.DEFLATE compression algorithms. The compression is applied before any IPSec modification or fragmentation is performed. It is often useless to compress random or already compressed data (e.g., .mp3 or .rar files); in fact, the extra compression applied sometimes results in the increase of the IP packet size. Besides, if you are using an IPSec tunnel over PPP or SLIP, they might compress data at the lower layer; so if IPComp is turned on, the overall communication performance will suffer, as the data will go through two compression processes.

The IPComp protocol introduces negotiation of an additional component to a successful operation. Before the endpoints are able to communicate, the IPComp Association (IPCA) must be established using the IKE mechanism. We have to mention that IPComp is flexible and you can selectively apply compression only to a specific transport layer protocol or to one end of the established connection.

IPSec Key Exchange and Management Protocol

IPSec Key Exchange and Management Protocol (ISAKMP) is part of the IPSec protocol suite that defines the procedures for negotiation, establishment, modification, and deletion of SAs, as well as the used packet format. It was designed to be independent from any specific key exchange or key generation techniques, cryptographic algorithms, or authentication mechanisms. ISAKMP defines a general framework and is rather abstract in its application. We focus in more detail on the IKE mechanism that is based on the ISAKMP framework.

IKE

Internet Key Exchange (IKE) is a general-purpose security exchange protocol that provides utility services for IPSec authentication of the IPSec nodes, negotiation of IKE and IPSec SAs, and establishment of keys for encryption algorithms. The specification of what IKE can be used for is defined in the Domain of Operation (DOI) RFC 2407.

IKE consists of two different modes that operate in one or two ISAKMP phases. Phase 1 is used for the establishment of a secure channel used later to protect all negotiations occurring in Phase 2. Essentially, ISAKMP SA is established after the negotiations between both ISAKMP peers. The following functions are performed during IKE Phase 1:

• Authentication and protection of IPSec nodes' identities

• Matching IKE SA policy negotiation to protect IKE exchanges

• Authenticated Diffie-Hellman exchange to establish a matching shared secret key

• Tunnel establishment for the IKE Phase 2 negotiation

Phase 2 is used to negotiate the IPSec SAs employed to set up an IPSec tunnel to protect IP traffic. A single Phase 1 SA can be used to negotiate Phase 2 SAs. The following functions are performed during IKE Phase 2:

• IPSec SA parameter negotiations

• IPSec SA establishment

• Periodic renegotiation of the IPSec SA

For the peers to be able to establish any form of secure communication between each other, a requirement for the initial authentication has to be satisfied. The typical IPSec implementation relies on the following methods:

• Preshared key (PSK). This authentication method relies on proof of possession of a shared secret between two parties eligible for communication. The factor with a potential to compromise the security of this solution is unsafe distribution of the PSK.

• Public Key Algorithm. This authentication method relies on public–private key pair generation. The public keys can be safely exchanged over the means of insecure communication, but the question of establishing true ownership of the key is arising.

• Digital certificates. The public key distribution scheme requires some level of trust. On networks such as the Internet, where control is highly questionable and the infrastructure is untrusted, the distribution of keys can be troublesome. The same applies to wireless networks, which are additionally susceptible to Layer 2 kracker_jack-style man-in-the-middle attacks. With the introduction of the third party, the acknowledged CA, digital certificates are issued containing the certificate bearer's identity, name or IP address, serial number, expiration date, and a public key. The standard digital certificate format is defined as X.509.

Phase 1 Modes of Operation

There are three possible ways to negotiate SA in Phase 1:

• Main mode. This mode was designed to separate key exchange information from the identity and authentication information to protect identity information under the previously generated Diffie-Hellman shared secret. This mode exchanges six UDP datagrams.

• Aggressive mode. This exchange mode allows the transmission of key exchange, identity, and authentication together. It is often used when the protection of the identity information is not important. Three UDP datagrams are exchanged. On receipt of the first proposal message, numerous resources are spent generating the response message. If several spoofed consequent proposal messages are sent, the consumption of significant resource power might occur, resulting in a DoS.

• Base mode. Four UDP datagrams are exchanged. This mode avoids the computationally intensive part of the Aggressive mode until the initiating party confirm its existence. It is supposed to accumulate the advantages of the Aggressive mode, but unless the parties use public key encryption, the identity data is not protected.

Phase 2 Mode of Operation

On the other hand, IKE Phase 2 has only one mode of negotiation, Quick Mode. It occurs after the IKE has successfully established a secure tunnel in mode 1; therefore, all the data used in negotiations is encrypted. The connection can be initiated by either peer and one or more IPSec SAs that are negotiated on the exchange of three messages by hosts.

Perfect Forward Secrecy

Another feature of IPSec that greatly enhances security is Perfect Forward Secrecy (PFS). When enabled, a new Diffie-Hellman exchange is performed for each Quick Mode. Therefore, if one of the ISAKMP SAs is compromised, it will not affect other SAs. The downside is that CPU usage is increased, negatively affecting the performance of such a system.

Dead Peer Discovery

There is an internal mechanism in IPSec that can be used to send a delete notification payload via IKE when the peer is disconnecting an IPSec SA. Unfortunately, the peer does not usually send this notification for a simple reason like power failure or system crash. This situation is frequently exhibited on wireless networks when the host suddenly leaves the coverage zone. The mechanism that tries to solve this problem is the Dead Peer Discovery (DPD) mechanism. It works by sending a notify payload prior to sending data if the period of communication inactivity is longer than some set value. If the peer is alive, the incoming notify payload is respected by returning one of its own.

IPSec Road Warrior

A typical situation involves a client connecting from a remote location or over a wireless link and getting a different IP address assigned by DHCP that changes from time to time. As one of the IP addresses is dynamic, it cannot be used to verify the identity of the peer. Therefore, an alternative way of authenticating such hosts has to be used (e.g., X.509 certs).

Opportunistic Encryption

The idea behind opportunistic encryption is to allow peers to communicate securely and without any prior knowledge of each other. Before a host sends out the packet, it checks whether it is possible to establish a secure link with the receiving party. If both machines are set up to understand the opportunistic encryption, a secure tunnel will be established. The method relies on DNS for distribution of RSA public keys presented on request. It is feasible for wireless hot spots, but it can be vulnerable to DNS spoofing and various man-in-the-middle attacks, including wireless-specific attacks on the data link layer (see the kraker_jack tool from the Airjack suite).