docker 101 @kacst saudi hpc 2016

Saudi HPC 2016 Tutorials@WALIDSHAARI21st April 2016

§ KACST to giving us this opportunity

§ Docker Inc. For allowing us use some of their graphics from docker slideshare

§ Christian Qnib @CQnib for the valuable HPC insights, and slide ideas for CNAME

§ All for attending

§ vim/Emacs

§ Linux

§ Git

§ vagrant

§ Windows

§ Development/Operations/other

§ Devops

§ Configuration management tool

§ Containers

§ Schedulers

§ Containers management tools, or distributions

§ Cluster management tools

0.0.3 March 2013

1.0 June 2014

1.1 July 2014

1.2 August 2014

1.3 October 2014

1.4 December

2014

1.5 February

2015

1.6 April 2015

1.7 June 2015

1.8 August 2015

1.9 November

2015

1.10 Feburary

2016

1.11 April 2016

Linux features?

Name spaces?cgroups?Lxc?

Union file systems

Configuration management ?

Virtualization technology?

npm

jar

Packaging ?

rpm

deb

tar.gz

Virtual/environment management ?

Manualconfiguration

Traditional VMs

Less PortableMinimal overhead

Most PortableLots of overhead

Configuration Management tools

Containers

Docker

Intel ClearContainers

§ There are other containers § Shifter

§ Didn’t like it, it made wrong assumptions about containers, and meant for CRAY, it exploits the current Docker registry system to build its own chrooted squashefscontainers in loopback devices, does not have the full Docker advantages.

§ Singularity§ Rocket

§ Back to the Linux/Unix philosophy, developed by CoreOS team

§ Intel clear containers§ One possible solution to Container security

Container encapsulate an application completely with all of its software dependencies into a standardized unit for software portable across different platforms*.

https://www.docker.com/what-docker

Bin/Libs

APP C

Docker Engine

Docker capable OS

Bin/Libs

App B

Bin/Libs

APP A

Bin/Libs

Database

Docker Engine

Docker capable OS

Bin/Libs

Load balancer

Bin/Libs

Web ServerWeb Server

Bin/Libs

•Open Source Devops platform • empowers developer to build and package services and

applications• ship containerized applications as independent building unit

“Lego”• Enables continuous delivery, and deployment• Portability, reproducibility• Behaves the same way in production, or other env.

• Run standalone or distributed

§ Docker Core Components:§ Engine : Portable (light/heavy) weight run time, and packaging tool§ Registry : Central repos to manage sharing of packaged Docker images, and

enabled (build, ship, release, run) workflows ( the default registry Docker HUB)

§ Containers Eco System§ Modular

§ Check Schedulers§ Compliant

§ OCI§ Holistic

§ End to end workflows

§ cgroups by Google

§ namespace by IBM

§ selinux by NSA

§ iptables

§ Docker in March 2013 was a lightweight runtime and robust tooling to build and ship the encapsulated operating environment to run virtual containers on Linux for your distributed applications.

§ Image:The basis of a Docker container

§ Docker Container: The standard unit in which the application service resides "running image instance"

§ Docker Engine: Runs on Linux, Creates, ships and runs Docker containers deployable on physical or virtual host locally, in a datacenter or cloud service provider

§ Docker Machine:Docker Machine is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with docker- machine commands. You can use Machine to create Docker hosts on your local Mac or Windows box, on your company network, in your data center, or on cloud providers like AWS or Digital Ocean.

§ Docker Registry: On-premises registry for image storing and collaboration

Containers

• Packages up software binaries and dependencies "minimal"• Isolates software from each other• Container is a standard format• Easily portable across environment• Allows ecosystem to develop around its standard Container

§ Deploy faster, and more often

§ Shorter time between deploys

§ Resource usage improved too (2x-10x)

§ Portability Freedom: Move containers around (laptop to workstation, laptop to servers, colo-to-cloud, cloud-to-cloud, cloud-to-colo, ...)

Network Storage I/OMemoryCPU

CGROUP 1

CGROUP 2

CGROUP 3

CGROUP 4

CGROUP 5

CGROUP 6 CGROUP 8

CGROUP 9

CGROUP 7

- Other subsystems:• DEVICE• HUGETLB• CPUSET

https://github.com/yadutaf/ctop

Host

Isolation is achieved via Linux kernel namespaces

Container 3pbs_serve

rpbs_sched

sshd

Container 2

mpi_task(s)

pbs_mom

sshd

Container 1

mpi_task(s)

pbs_mom

sshd

Mount IPC UTSNetworkPID USER

Container have several name spaces

Host

Container 2

mpi_task(s)

pbs_mom

sshd

Container 1

mpi_task(s)

pbs_mom

sshd

Mount IPC UTSNetworkPID

Container 3pbs_serve

rpbs_sched

sshd

USER

Network namespace can be shared between one or more container

Host

Container 2

mpi_task(s)

pbs_mom

sshd

Container 1

mpi_task(s)

pbs_mom

sshd

Mount IPC UTSNetworkPID

Container 3pbs_serve

rpbs_sched

sshd

USER

https://docs.docker.com/

1- Go to https://docs.docker.com/windows/2- then start installation https://docs.docker.com/windows/step_one/3- Ensure git bash terminal is installed4- invoke a git bash terminal5- verify the docker-machine

1- Initialize the Docker machine environment2- Verify the Docker-machine

docker-machine ls

docker-machine env default

3- You might need to create a virtual default machine

You might need to create a virtual default machinedocker-machine create --driver virtual box default

eval `docker-machine env default`docker run hello-world

docker search hello-world

docker search –s 20 alpine

curl -s -S 'https://registry.hub.docker.com/v2/repositories/library/alpine/tags/' | jq '."results"[]["name"]' |sort

docker pull docker.io/seqvence/static-sitedocker pull mhart/alpine-nodedocker pull python:2.7-alpinedocker pull manomarks/workerdocker pull redis:alpinedocker pull postgres:9.4docker history syed/linpack

$docker pull alpine:2.6$docker images

https://medium.com/@mccode/the-misunderstood-docker-tag-latest-af3babfd6375#.2rn04fqc9

$docker run alpine:latest cat /etc/alpine-release$docker run alpine:2.6 cat /etc/alpine-release$docker run alpine:2.6 uptime$docker run alpine:latest uptime

$docker history alpine:latest$docker history alpine:2.6$docker history syed/linpack

docker images

docker ps -a

-I interactive-t

for i in {1..10}; do docker run alpine uptime; done

docker ps -a

man docker-rm

docker ps –a

docker rm container [container...]

Or:

docker rm `docker ps --all -q`

What if you want to remove:

volumes associated with the container?

Links associated with the container?

Running container(s)?

docker stats

docker ps -a

[walid@silverII ~] $ docker run seqvence/static-site

How to pass a signal to Kill?

docker run --name static-site -e AUTHOR="Walid Shaari" -d -P seqvence/static-site

§ From the docker-machine§ $ docker-machine ip default

docker run --name static-site-2 -e AUTHOR="Your Name" -d -p 8888:80 seqvence/static-site

§ Base

§ Child

§ Official

§ User

§ https://github.com/docker/docker-birthday-3

§ In your homedir§ git clone https://github.com/docker/docker-birthday-3§ cd docker-birthday-3/example-voting-app/§ Follow https://github.com/docker/docker-birthday-3 instruction for compose, and swarm§ Check example enteries

§ https://blog.docker.com/2016/04/docker-birthday-app-challenge-winners/

https://github.com/yadutaf/infilter

To share the host network, PID, and IPC namespaces with the container,

run the following command:$ docker run -it --net=host --pid=host --ipc=host alpine sh

• There is more• Schedulers

• Nomad• Kubernetes• Mesos

• Storage engines• ZFS• BTRFS• CEPH

• Network overlays• And definitely there is more to be awesome!