RED HAT ENTERPRISE LINUX 7 1

Checkpoint Restore In Userspace

SONG, CHANGAN(Leo)APAC Technical Account Manager,Customer Experience & Engagement,Strategic Customer Engagement, Red Hat

RED HAT ENTERPRISE LINUX 7 2

• 프로세스의 현재 상태 저장• 이전 상태 복원 기능 (checkpoint 전으로 )• Checkpoint 된 프로세스의 모든 정보는 하나이상의 이미지 파일로 저장

됨 ( 저장정보 : memory pages, file descriptors, inter-process

communication, and so on)• 같은 시스템 또는 다른 시스템에 프로세스 복원• 컨테이너 라이브 마이그레이션 같은 용도로 사용됨• RHEL7.3 에 패키지가 포함 (criu 2.3)• Tech-preview 기능으로 등록

CRIU

Checkpoint / Restore In User space

https://access.redhat.com/articles/2455211

RED HAT ENTERPRISE LINUX 7 3

CRIU

how does it works?

Kernel objects Process tree

criu

Image files

Namespaces

Files

Sockets

Pipes

001101101010110001011010000011010101

001101101010110001011010000011010101

001101101010110001011010000011010101

001101101010110001011010000011010101

001101101010110001011010000011010101

001101101010110001011010000011010101

RED HAT ENTERPRISE LINUX 7 4

CRIU

how does it works?

Kernel interfaces

Dump Restoresyscalls

netlink

/proc/

ptrace

RED HAT ENTERPRISE LINUX 7 5

CRIU

Dump Parasite code

Receive file descriptors

Dump memory content

Prctl(), sigaction, pending signals, timers, etc.

Ptrace

freeze processes

Inject a parasite code

Netlink

Get information about sockets, netns

Procfs

/proc/PID/maps, /proc/PID/map_files/, /proc/PID/status, /proc/PID/mountinfo

RED HAT ENTERPRISE LINUX 7 6

CRIU

Restore

Collect shared objects

Restore name-spaces

Create a process tree

Restore SID, PGID

Restore objects, which should be inherited

Files, sockets, pipes, ...

Restore per-task properties.

Restore memory

Call sigreturn

Awesome

Namespaces

Processes

RED HAT ENTERPRISE LINUX 7 7

CRIU

Interest moment

How to restore shared objects?

Send file descriptors via unix sockets

Map files from /proc/self/map_files/ for restoring anon shared mappings

How to restore memory mappings on the correct places?

Map a new code block and a stack

Unmap crtools' mappings

Remap task's mappings on the correct places

How to resume a process?

Create a signal frame

Call sigreturn()

RED HAT ENTERPRISE LINUX 7 8

• HPC 환경을 위해 개발• 하나의 어플리케이션이 수백 , 수천 코어에 분산되어 실행되는 환경에

적합• 특히 어플리케이션이 실패할 경우 , 전체 CPU 사용된 것이 쓸모없게

되고 데이터도 손실되는 약점을 CRIU 로 해소

• 어플리케이션과의 호환성 검토 필요

• 초기에는 관심받지 못하다가 container migration 으로 각광

CRIU

Birth of CR

RED HAT ENTERPRISE LINUX 7 9

• Inter-process-communication(IPC) 을 이용하여 checkpoint /restore 동작이 가능 .

• 항상 부모 프로세서와 모든 자식 프로세서 checkpoint/restores 에 대해서 가능 .

• PID 항상 같아야 하며 , 시스템에서 이미 사용하는 PID 가 있는 경우 , CRIU 를 이용한 프로세스 복구 단계에서 실패 .

CRIU

Limitations

https://criu.org/What_cannot_be_checkpointed

RED HAT ENTERPRISE LINUX 7 10

Live migration

CRIU

Host A Host B

Shared FS

Pre-migrate memory

with memory tracker

http://criu.org/P.Haul

RED HAT ENTERPRISE LINUX 7 11

Load balancing on cluster

CRIU

Host A

Host C

Host B

RED HAT ENTERPRISE LINUX 7 12

Power saving on cluster

CRIU

Host A

Host C

Host B

RED HAT ENTERPRISE LINUX 7 13

Node maintenance

CRIU

Host A Host B

RED HAT ENTERPRISE LINUX 7 14

Kernel upgrade w/p reboot

CRIU

Host

Kernel A

KexecKernel B

RED HAT ENTERPRISE LINUX 7 15

Slow services startup

CRIU

time# service foo start

Service readiness

Spawn process

Load config

Top-up caches

Initialize resource pools

Ready

T

100%

RED HAT ENTERPRISE LINUX 7 16

Slow services startup

CRIU

time

Tt < T

Ready

Spawn process

100%

Service readiness

# service foo restore

RED HAT ENTERPRISE LINUX 7 17

Periodic snapshots

CRIU

time

Memory tracker helpsto keep images smaller

RED HAT ENTERPRISE LINUX 7 18

HPC

CRIU

time

Powerfailure

0% 20% 40% 60% 60%

RED HAT ENTERPRISE LINUX 7 19

Advanced debugging

CRIU

Production Host

Applicationin trouble

Developer Host

Debugger

RED HAT ENTERPRISE LINUX 7 20

Advanced testing

CRIU

...

New testor

new hardware ?

RED HAT ENTERPRISE LINUX 7 21

Installation ciru package on RHEL7

CRIU

# yum install criu -y...Dependencies Resolved============================================================================================= Package Arch Version Repository Size=============================================================================================Installing: criu x86_64 2.3-2.el7 rhel-7-server-rpms 349 kInstalling for dependencies: protobuf-c x86_64 1.0.2-3.el7 rhel-7-server-rpms 28 k…

# ldd `which criu`linux-vdso.so.1 => (0x00007ffed554d000)librt.so.1 => /lib64/librt.so.1 (0x00007f5fd0faf000)libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5fd0d93000)libprotobuf-c.so.1 => /lib64/libprotobuf-c.so.1 (0x00007f5fd0b89000)libdl.so.2 => /lib64/libdl.so.2 (0x00007f5fd0985000)libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f5fd0764000)libc.so.6 => /lib64/libc.so.6 (0x00007f5fd03a2000)/lib64/ld-linux-x86-64.so.2 (0x00007f5fd11bd000)libm.so.6 => /lib64/libm.so.6 (0x00007f5fd00a0000)

RED HAT ENTERPRISE LINUX 7 22

1) criu on command

CRIU

How to Use

2) criu in runc

- restore checkpoint container

http://rhelblog.redhat.com/2016/12/08/container-live-migration-using-runc-and-criu/

# criu --help

Usage: criu dump|pre-dump -t PID [<options>] criu restore [<options>] criu check [--feature FEAT] criu exec -p PID <syscall-string> criu page-server criu service [<options>] criu dedup...

# runc checkpoint <container name> For example, # runc checkpoint rhel7-httpd

# runc restore -d <container name>For example,# runc restore -d rhel7-httpd

- store checkpoint container

RED HAT ENTERPRISE LINUX 7 23

Demo in runc

CRIU

RED HAT ENTERPRISE LINUX 7 24

Runc

CRIU

criu can now be used for following applications running in a Red Hat Enterprise Linux 7 runc container:

vsftpd apache httpd sendmail postgresql mongodb mariadb mysql tomcat dnsmasq

RED HAT ENTERPRISE LINUX 7 25

RED HAT ENTERPRISE LINUX 7 26

THANK YOU