\svnInfo $Id: paper.tex 2895 2008-05-28 22:43:44Z /C=US/ST=California/L=San Francisco/O=Ambulatory Clam Network/OU=Certification Authority/CN=Tal Garfinkel/emailAddress=talg@vmware.com $

The complexity of commodity operating systems brings with it the
presence of vulnerabilities. Consequently, a great deal of work has
studied how to mitigate the impact of a compromise by protecting OS
components or applications through mechanisms such as microkernels,
virtual machine monitors, and new processor architectures.
Unfortunately, most work has focused on CPU and memory isolation and
neglected OS semantics.  Thus, while we know much about how to
prevent OS and application processes from directly modifying each
other, far less is understood about how different OS components
can undermine application security if they turn malicious.

We consider this problem in in the context of our own work on
Overshadow, a virtual-machine-based system for retrofitting
protection in commodity operating systems. We explore how malicious
behavior of each major OS subsystem can potentially undermine
application security, and propose measures for mitigating such
behavior. While our discussion is presented in terms of Overshadow
and Linux, many of the problems and solutions are applicable to
other system attempting to cope with the problem of a malicious OS.