Kenichi Kourai (Kyushu Institute of Technology)Takuya Nagata (Kyushu Institute of Technology)

A Secure Framework forMonitoring Operating SystemsUsing SPEs in Cell/B.E.

OSes are not an exception of attacks e.g. kernel rootkits All the applications are also compromised if the

OS is compromised It is necessary to check the integrity of

OSes Not only at the boot time, but also at runtime

OSes are long-running software This can increase the reliability of the system

Attacks against OSes

kernelrootkit

OS

application

application

Running on top of the OS Monitoring systems must issue system calls to

the OS e.g. obtaining process information e.g. reading the kernel memory

The results cannot be trusted if the OS is compromised

Embedded into the OS Monitoring systems can directly examine the

kernel They are easily disabled by the compromised

OS

Secure OS Monitoring is not Easy

OS

monitoring

system

monitoring

system

system calls

Two approaches have been proposed The underlying hypervisor monitors the OS in a

virtual machine (VM) A privileged VM monitors the OS in a target VM

The hypervisor and the privileged VM can be also compromised There are vulnerabilities in software

VM-based Approaches

hypervisor

OS

monitoring

system

monitoring

system

privilegedVM

targetVM

Using System Management Mode (SMM) in x86 One of many hardware-based approaches A CPU can securely execute a monitoring

system in SMM A monitoring system is located in isolated

SMRAM Several drawbacks

SMM is much slower than the normal mode A monitoring system must be embedded in

BIOS

Hardware-based Approaches

monitoring

system

SMRAMSMM

mainmemory

normalmode

CPU

A framework for securely monitoring OSes using Cell/B.E. Runs a monitoring system on an SPE

An SPE is a general-purpose CPU core Its isolation mode enables secure execution

Monitors the running status of the monitoring system from an external security proxy

SPE Observer

PPE

OS

SPE

monitoring

system

security

proxy

targethost

Cell/B.E.

Heterogeneous multicore processor PPE (control processing core)

Runs the OS and regular processes SPE (arithmetic processing core)

Runs parallel applications Contains the memory called a local store Accesses the main memory using DMA

Architecture of Cell/B.E.

SPE SPE SPE

SPE SPE SPE

PPE

mainmemor

ylocalstore

DMA

DMA

Protects the local store in an SPE from the PPE and the other SPEs Preserving integrity

Attackers cannot modify a running monitoring system or processing data

Preserving confidentiality Attackers cannot analyze a monitoring system or

steal sensitive information

Isolation Mode

isolated SPE

monitoring

system

local store

PPE SPE

OS application

Securely loads a monitoring system into the local store of an isolated SPE Preserving integrity

Attackers cannot load compromised images of monitoring systems

Preserving confidentiality Attackers cannot decrypt images of monitoring

systems

Secure Loader

mainmemory

encrypted

imageencrypte

dimage

monitoring system

DMA

isolated SPE

DMA secure loader

verify & decrypt

verify & decrypt

The isolation mode is not perfect for secure execution of monitoring systems The PPE can stop the execution of even

isolated SPEs It must control all the SPEs

Attackers can disable monitoring systems! The isolation mode is not designed for PPE

monitoring Fortunately, the confidentiality of monitoring

systems is still preserved

Availability Issue

PPE

OS

isolated SPE

monitoringsystem

stop

Externally monitors the running status of monitoring systems on SPEs Periodically sends heartbeats to monitoring

systems via the relay process Cuts the network if monitoring systems do

not respond to the heartbeats correctly

Security Proxy

PPE SPE

monitoring

systemsecurit

yproxyexternal

network

heartbeats relayprocess

target host

OSinternalnetwork

The security proxy sends an encrypted challenge to a monitoring system The monitoring system decrypts it and returns

an encrypted response Attackers cannot return correct responses

Only legitimate monitoring systems and the security proxy share secret keys

A malicious relay process cannot mount man-in-the-middle attacks

Secure Heartbeats

securityproxy

monitoring

system

relayprocess

encryptedchallenge

encryptedresponse

SPE Observer can schedule monitoring systems Application performance is improved by not

occupying one SPE for a monitoring system Scheduling by the security proxy and the SPE

scheduler The security proxy periodically sends commands The OS schedules SPEs if necessary

Scheduled Monitoring

PPE

OS

SPE

security

proxy

commandsrelay

process

SPE SPE

application

...

...monitorin

gsystem

Integrity monitor for the OS kernel Obtains the contents of the kernel memory

using DMA Calculates SHA-1 hash and compares it with

correct one Overlaps DMA with calculation using double

buffering Other possible monitors

Monitors for dynamic kernel data Using a technique similar to VM introspection

Examples of Monitoring Systems

PPE

OS kernel

SPE

integritymonitor

DMA

SHA-1

DMA

buffers

DMA

SHA-1

SPE Observer configures an isolated SPE to enable accessing the kernel memory Clears the Problem-State bit in the status

register of the MFC The MFC is used for DMA transfers

Registers an address mapping for the kernel memory to the SLB

The SLB is an address translation table

Accessing the Kernel Memory

local store

MFC SLBkernel memory

mainmemory

SPE

DMA

We conducted several experiments to examine Effectiveness and performance of the integrity

monitor Impacts on application performance

We used the emulation of the isolation mode because we could not obtain the secure loader

supporting the hardware-level isolation mode

Experiments

CPU: 1 PPE, 6 SPEsLocal store: 256 KBMain memory: 256 MBNIC: Gigabit EthernetOS: Linux 2.6.27

CPU: Xeon E5630Memory: 4 GBNIC: Gigabit Ethernet

PlayStation 3 Security proxy

We ran the integrity monitor on an SPE It could detect the compromised kernels

Modified system call table Modified function for a system call

We measured the time forintegrity check Hash calculation: 70% DMA was hidden by

calculation

Integrity Check of the Kernel

0

5

10

15

20

25

30

hash calcula-tion

tim

e (

ms)

We ran various applications with various monitors CPU- and DMA-bound applications

Using various numbers of SPEs CPU- and DMA-bound monitors

Using one SPE

Impacts on Application Performance

mainmemory

SPE SPE

monitoring

system

SPE SPE SPE SPE

CPU-boundapplication

DMA

DMA-boundmonitor

We ran various applications with various monitors Any monitors did not affect CPU-bound

applications Linear performance improvement

All monitors affected DMA-bound applications Especially DMA-bound monitors

Impacts on Application Performance

0 1 2 3 4 50

2

4

6

CPU-bound Application

CPU-bound monitor

# of SPEs

imp

rovem

en

t

0 1 2 3 4 50.0 0.5 1.0 1.5 2.0 2.5

DMA-bound Application

DMA-bound m...

# of SPEs

imp

rovem

en

t

We ran various applications using 6 threads with the integrity monitor The monitor occupied one SPE

An application could use only5 SPEs

Performance Degradation

SPE

integritymonitor

SPE

thread1

application

SPE

thread2

SPE

thread3

SPE

thread4

SPE

thread5

thread6

We ran various applications using 6 threads with the integrity monitor The monitor occupied one SPE

An application could use only5 SPEs

Application performance CPU-bound: 83% (= 5/6) DMA-bound: 98%

DMA bandwidth was saturated Matrix: 18%

Waiting for synchronization

Performance Degradation

0.0

0.2

0.4

0.6

0.8

1.0

CPU-bound DMA-boundMatrix

perf

orm

an

ce

SPE Observer scheduled the integrity monitor at various intervals We measured the performance of matrix

multiplication

Improvement by Scheduling (1/2)

SPE

integritymonitor

SPE

thread1

application

SPE

thread2

SPE

thread3

SPE

thread4

SPE

thread5

thread6

SPE Observer scheduled the integrity monitor at various intervals We measured the performance of matrix

multiplication

The performance wasimproved as the interval became

longer 83% at a 200-ms

interval = 5/6

Improvement by Scheduling (1/2)

0 200 400 600 800 10000

0.2

0.4

0.6

0.8

1

Matrix

scheduling

interval (ms)

perf

orm

an

ce

We measured the performance of CPU- and DMA-bound applications CPU-bound: 96% at a 100-ms interval DMA-bound: almost 100% at a short interval

Improvement by Scheduling (2/2)

0 200 400 600 800 10000.0

0.2

0.4

0.6

0.8

1.0

CPU-bound Application

scheduling

interval (ms)

perf

orm

an

ce

0 200 400 600 800 10000.80

0.85

0.90

0.95

1.00

DMA-bound Application

scheduling

interval (ms)

perf

orm

an

ce

Copilot [Petroni et al.'04] Sends the contents of the physical memory to

a remote host using a special PCI card The remote host checks the integrity of the OS

Flicker [McCune et al.'08] Executes security-sensitive code using Intel

TXT The whole system is suspended while such

code is running Code verification service [Murase et al.'09]

An isolated SPE checks the integrity of applications for running on the PPE

Not assume that the OS is compromised

Related Work

We proposed SPE Observer A framework for secure execution of OS

monitoring systems Using the isolation mode of SPEs

to guarantee the integrity and confidentiality Using the security proxy

to monitor the running status of monitoring systems

Scheduling monitoring systems to mitigate performance degradation

Future work Developing various monitoring systems Developing middleware for better SPE

scheduling

Conclusion