Zero-copy Migrationfor Lightweight Software

Rejuvenationof Virtualized Systems

Kenichi KouraiHiroki Ooba

Kyushu Institute of Technology

Software Aging [Huang+ FTC'95]

Virtualized systems tend to suffer from software agingThe state of running software is degraded with

timeE.g., memory leakage

Hypervisors (and management VMs) are long-running software

Source: F. Machida et al., Combined Server Rejuvenation in a Virtualized Data Center, Proc. IEEE ATC 2012.

free memory free disk space

Software Rejuvenation [Huang+ FTC'95]

Restore systems to the normal stateProactive technique for counteracting software

agingSimplest method: system reboot

Cause a long downtime in virtualized systemsNeed to stop all VMs during the rebootViolate service level agreement (SLA)

aged hypervisor

VM VM...

Rejuvenation with VM MigrationReduce downtime during rejuvenation

Migrate all VMs to another hostThe downtime due to VM migration is usually

negligibleReboot only the aged hypervisor

No VMs on it

aged hypervisor

VM...

clean hypervisor

migration

source host destination host

VM VM...VM

Performance DegradationVM migration stresses hosts and network

largelyTransfer the memory images of VMs via network

Several hundreds of GB in totalEncrypted to prevent eavesdropping/tampering

Occupy CPUs and memory/network bandwidthsDegrade the performance of virtualized systems

Source: K. Kourai et al., Fast Software Rejuvenation of Virtual Machine Monitors, TDSC, 2011.

web throughputstart end

VMBeamEnable lightweight software rejuvenation

Start a new virtualized system at the same hostUsing nested virtualization

Migrate all VMs from an aged system onto a clean oneUsing zero-copy migration

Stop the aged system

aged hypervisor

VM... zero-copymigration

source virtualized system destination virtualized system

clean hypervisor

VM VM...VM

Nested VirtualizationEnable a virtualized system to run in a VM

Guest hypervisor/VMs inside a virtualized system

Host hypervisor/VMs in the outsideThe overhead is 6-8% [Ben-Yehuda+ OSDI'10]

1% in a special-purpose host hypervisor [Tan+ DCDV'12]

guest hypervisor

guestVM

guestVM

...

guest hypervisor

guestVM

guestVM

...

host hypervisor

host VM host VM

Zero-copy MigrationRelocate the memory of guest VMs

between virtualized systems at the same hostStep 1: Share the memory between src/dst

guest VMsThe src guest VM can continue to run

Step 2: Release the memory of the src guest VMAfter the entire memory is shared

clean guest hypervisor

host hypervisor

inter-guestmemory sharing

clonedguest VM

runningguest VM

aged guest hypervisor

destinationhost VM

sourcehost VM

No Memory Re-transferZero-copy migration is completed in one

iterationNot repeat to re-transfer modified memory

areasTraditional live migration needs multiple iterations

Modifications are directly reflected to a destination guest VM by memory sharing

Reduce the migration time for memory-intensive VMs

clean guest hypervisor

host hypervisor

aged guest hypervisor

nore-transfer

clonedguest VM

runningguest VM

destinationhost VM

sourcehost VM

Reducing System LoadsNo use of the virtual network

Shared memory is used

No copy of large memory images of VMsThe memory is simply relocated

No encryption of the memory imagesAny data is not exposed to the outside of guest

VMs

No need to detect memory write in guest VMsModifications are directly reflected

CPU

Net

MemCPU

CPU

CPU

Devirtualization [Lowell+ ASPLOS'04]

Remove the overhead of nested virtualizationDisable the host hypervisor during a normal runRe-virtualize the system only during

rejuvenationCons: the guest hypervisor could directly

corrupt the hardware state

guest hypervisor

guestVM

...

host hypervisor

host VM host VM

guest hypervisor

guestVM

...

host hypervisor

host VM

devirtualize

revirtualize

<normal run> <software rejuvenation>

Isn't the Host Hypervisor Aged?Yes, but the aging speed is slower

Much smaller than the guest hypervisor6K LOC (CloudVisor) vs. 300K LOC (Xen 4.2)

Execute no complex VM operationsDevirtualization can suppress aging

The host hypervisor is disabled

minimal host hypervisor

feature-richguest hypervisor

host VMguestVM

migration

ExperimentsWe confirmed the effectiveness of zero-

copy migration in VMBeamSystem loads, migration time, and downtime

ComparisonXen-Phys

Traditional system with two physical hostsXen-Blanket [Williams+ EuroSys'12]

System with nested virtualization and fast virtual network[2 hosts]CPU: Intel Xeon E5-2665Memory: 32 GBNIC: Gigabit Ethernet

host hypervisor: Xen 4.2host Dom0 OS: Linux 3.2.0guest Dom0 OS: Linux 3.5.0

System LoadsWe measured system loads during VM

migrationVMBeam did not transfer data via virtual

networkIt used only 30% of CPU time in Xen-PhysIt did not access the VM memory (estimated)

CPU0

5

10

15

20

25

30

Xen-Phys (src) Xen-Phys (dst) VMBeam Xen-Blanket

tim

e (

10

3 s

ec)

Network0

1

2

3

4

5

tran

sfe

r (G

B)

Memory0

10

20

30

40

50

60

access (

GB

)

Migration PerformanceWe measured the migration time and

downtimeThe migration time in VMBeam was up to 5.8x

fasterThe downtime in VMBeam was 0.2s longer

Due to the overhead of nested virtualization

0 1 2 3 40

30

60

90

120

150

Xen-Phys VMBeam Xen-Blanket

memory size (GB)

mig

rati

on

tim

e

(sec)

0 1 2 3 40

0.5

1

1.5

memory size (GB)

dow

nti

me (

sec)

16s

Related WorkMicrovisor [Lowell+ ASPLOS'04]

Maintain the system in a new VM and migrate applications to it

Focus on devirtualizationRDMA-based migration [Huang+ Cluster'07]

Only one copy by InfiniBandNeed 3 copies when encrypting the memory

imageWarm-VM Reboot [Kourai+ DSN'07]

Maintain VMs in memory during rejuvenationStill cause downtime during the hypervisor

reboot

ConclusionVMBeam for lightweight software

rejuvenation of virtualized systemsNested virtualization: Run aged and clean

systems at the same hostZero-copy migration: Migrate guest VMs

efficientlySuppress system loadsMake VM migration up to 5.8x faster

Future workDevelop a minimal host hypervisorEnable devirtualization in the host hypervisor