Data Execution Prevention in WindowsXP Service Pack 2

written by: Greg Melton; article published: year 2006, month 11;


In: Root » Computers and technology » Microsoft OS family » Data Execution Prevention in WindowsXP Service Pack 2

Dutch French Spanish Portuguese Italian German Japanese Chinese Korean Russian Arabic Bookmark and Share this Article

Starting with Windows XP Service Pack 2, Windows XP supports a feature called Data Execution Prevention (DEP) that prevents programs from replacing the original, intended machine instructions in memory with new instructions that could perform malicious acts. This feature became necessary when virus writers and hackers began exploiting bugs in software that can result in malicious program instructions sent from the attacker being written in memory that was supposed to hold just program data. These are often referred to as buffer overrun exploits or stack overflow exploits. If the program's path through memory eventually takes it into the malicious code, the injected instructions can do anything you can do; that is, the program has your privileges, can access any file you can, can infect other programs, and so on.

Data Execution Prevention uses two different mechanisms to guard against this type of attack: First, it uses Windows software mechanisms to prevent programs from writing any new instructions into sections of memory that were originally designated as holding instructions. Second, it prevents programs from executing instructions from any section of memory that was originally designated as holding only data. The second form of protection is the stronger of the two, but it is only available with some CPU chips, including all 64-bit processors from AMD and Intel, Intel's Pentium D and Pentium 840 Extreme Edition processors, and AMD's Sempron processors. This second mechanism is called Execute Disable or ED by Intel, and No Execute or NX by AMD, and it's used when available whenever Data Execution Prevention is enabled in Windows.

Note

On a corporate network, Data Execution Prevention is probably enabled and managed by the network Group Policy. Individual applications that are known to be safe but which modify their own executable instructions on purpose can be marked to "opt out" of protection using the Application Compatibility Toolkit. For more information about this mechanism, see www.microsoft.com/windows/appcompatibility/default.mspx.


By default, when Windows XP Service Pack 2 is installed, DEP is enabled only for Windows components themselves. To protect all applications, right-click My Computer and select Properties to open the System Properties dialog. Select the Advanced tab, and click the top Settings button under Performance. Select the Data Execution Prevention tab

To enable DEP for all applications, select Turn On DEP for All Programs and Services Except Those I Select.

If you change Data Execution Prevention settings, you'll need to restart Windows. When enabled for all applications, you may find that an application that used to work suddenly fails with a dialog box that says "Data Execution PreventionA Windows security feature has detected a problem and closed this program." In this case, you should contact the manufacturer's tech support to see whether this is a known issue, or if an update is available.

If you determine that the application is actually safe but just happens to require the ability to write modified instructions in order to work, you can instruct Windows to disable DEP for this application. Back in the DEP setup dialog click Add, and then browse to select the .EXE file that corresponds to the application in question. Click OK to save it in the list of exceptions.

Caution

If you enable Hardware Data Execution Prevention and have a flaky device driver, the driver may prevent Windows from booting. Use the following procedure to recover.

If Windows halts with a blue screen, or reboots repeatedly when you restart it after enabling Hardware DEP (or after updating a device driver when Hardware DEP is enabled), one of your device drivers is executing code from "No Execute" memory and terminating. Use one of the following methods to disable DEP. First, try to boot Windows in Safe mode:

1.
When your computer's BIOS startup screen appears, press F8 repeatedly until Windows' Advanced Startup Options menu appears. Select Safe Mode and press Enter.

2.
When Windows has started, log on as a Computer Administrator, go back to the Data Execution Prevention setup tab, and disable hardware protection. Restart Windows to test.

If Windows won't even boot in Safe mode, you'll need to take the more drastic step of manually editing the boot.ini file on the hard drive that contains Windows. To do this, follow these steps:

1.
Remove the hard drive from your computer and install it in another computer, which, if your Windows partition uses NTFS formatting, must be running Windows XP or Windows 2000. If you need to change the drive's master/slave jumpers, be sure to make a note of the original setting before changing them.

2.
Start up the alternative computer, and view My Computer to identify the drive letter that was assigned to your relocated drive; let's say it's E. (If a different drive letter is assigned, use that letter instead of E in the next step.)

3.
Open a command prompt window and type the following commands:

e:  attrib -r -h -s boot.ini  notepad boot.ini  

4.
In Notepad, locate the line under [operating systems] that has /NoExecute=OptIn, /NoExecute=OptOut or /NoExecute=AlwaysOn in it. Carefully change it to read /NoExecute=AlwaysOff.

5.
Save boot.ini (Alt+F, Alt+S) and close Notepad (Alt+X).

6.
Type the following command:

attrib +r +h +s boot.ini  

7.
Shut down the computer, remove your hard drive, reset the master/slave jumpers if you changed them, put it back in your computer, and restart Windows.

When Windows boots successfully, log on as a Computer Administrator and check the Event Log for an indication of which driver failed during startup. Update it or roll it back before enabling hardware DEP again.

Disclaimer

1) E-articles is not responsible for the information contained by this article as well for any and all copyright infringements by authors and writers. E-articles is a free information resource. If you suspect this article for any copyright infringement, please read the terms of service and contact us to investigate the problem.
2) E-articles is not responsible for inaccuracies, falsehoods, or any other types of misinformation this article may contain and will not be liable for any loss or damage suffered by a user through the user's reliance on the information gained here.

link to this article