kubernetes on aws

Pre-reqs:● Git: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git

● AWS CLI: http://docs.aws.amazon.com/cli/latest/userguide/installing.html

● kubectl:http://cs-k8s-workshop.s3.amazonaws.com/kubectl/darwin/amd64/kubectl

http://cs-k8s-workshop.s3.amazonaws.com/kubectl/linux/amd64/kubectl

● Bash

● git clone https://github.com/ContainerSolutions/kubernetes-aws-workshop

www.container-solutions.com | info@container-solutions.com

Kubernetes on AWSGrant Ellisgrant.ellis@container-solutions.com

Who’s who

● Presenters

● You!

➔ Developers? Ops? DevOps?

➔ Tools, languages & frameworks?

➔ Familiar or using any orchestration platform? Mesos/Swarm/ECS?

Purpose of the Workshop

● Get an overview of the components in kubernetes

● See how kubernetes leverages features present in AWS

● Get an idea of how a production setup may take shape

Scope of the Workshop

● Basic features of Kubernetes

● Brief look at AWS CloudFormation and IaaS components

● Hands on

Kubernetes

● From the Greek meaning “Helmsman” or “Pilot”

● Founded by Joe Beda, Brendan Burns and Craig McLuckie

● First announced by Google in 2014

Basic concepts

● Pods● Labels / Selectors● Replication Controllers / Replica Sets● Deployments● Services

All Resources can be expressed as YAML or JSON files

Pods● A pod is one or more containers● Ensures co-location / shared fate● Pods are scheduled, then do not move between nodes● Containers share resources within the pod:

➔ Volumes➔ Network / IP➔ Port space➔ CPU / Memory allocations

Pod exampleapiVersion: v1kind: Podmetadata: labels: name: influxdb name: influxdbspec: containers: - image: docker.io/tutum/influxdb:latest name: influxdb ports: - containerPort: 8083 name: admin protocol: TCP - containerPort: 8086 name: http protocol: TCP

Labels / Selectors

● Labels are arbitrary metadata● Attachable to nearly all API objects

➔ e.g.: Pods, ReplicationControllers, Services...● Simple key=value pairs● Can be queried with selectors

Labels example

- release=stable, release=canary- environment=dev, environment=qa, environment=prod- tier=frontend, tier=backend, tier=middleware- partition=customerA, partition=customerB- etc…

Labels example

Selectors explained

Labels are queryable metadata - selectors can do the queries:- Equality based:

- environment = production- tier != frontend- combinations: tier != frontend, version = 1.0.0

- Set based:- environment in (production, pre-production)- tier notin (frontend, backend)- partition or !partition

Selectors example

Replication Controllers

● Define the number of replicas of a pod● Will scheduled across all applicable nodes● Can change replica value to scale up/down● Which pods are scaled depends on RC selector● Labels and selectors are used for grouping● Can do quite complex things with RCs and labels

Example Replication ControllerapiVersion: v1kind: ReplicationControllermetadata: name: nginxspec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80

Replica Set

Replica Set is the next-generation Replication Controller. The only difference between a Replica Set and a Replication Controller right now is the selector support. Replica Set supports the new set-based selector which allow filtering keys according to a set of values:

- In- Notin- exists (only the key identifier)

For example:environment in (production, qa)

tier notin (frontend, backend)

partition

!partition

Deployments

A Deployment is responsible for creating and updating instances of your application

● Create a Deployment to bring up Pods and a replica set.

● Check the status of a Deployment to see if it succeeds or not.

● Later, update that Deployment to recreate the Pods (for example, to use a new image).

● Rollback to an earlier Deployment revision if the current Deployment isn’t stable.

● Pause and resume a Deployment.

Deployment exampleapiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginx-deploymentspec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.91 ports: - containerPort: 80

Services

“defines a logical set of Pods and a policy by which to access them”

● As Pods are ephemeral, we can't depend on Pod IPs

● Services find pods that match certain selection criteria

● Services can load balance between multiple Pods

● Services can have a single IP that doesn’t change

Services

A group of pods that act as one == Service- group == selector

Defines access policy- LoadBalanced, NodePort

Gets a stable virtual IP and Port- Called the service portal- Also a DNS name- On prem additional loadbalancer is needed

VIP is captured by kube-proxy- Watches the service consistency- Updates when backend changes

Service example

apiVersion: v1kind: Servicemetadata: name: railsappspec: type: NodePort selector: app: railsapp ports: - name: http nodePort: 36000 port: 80 protocol: TCP

Architecture

etcd (stores cluster state)API ServerSchedulerController managerKubelet (“node agent”)Kube-proxyContainer Runtime

https://github.com/kubernetes/kubernetes/blob/release-1.3/docs/design/architecture.md

ArchitectureMaster Node (“Control Plane”)

Api server- Point of interaction with the cluster- Exposes an http endpoint

Controller Manager- Responsible for most of the important stuff- Interacts with the api server to retrieve cluster state- Responsible for configuring networking- Allocates node CIDRs- Ensures correct number of pods are running- Reacts to Nodes being added / deleted- Manages Service Accounts and security tokens

Scheduler - Schedules newly created pods to a Node

ArchitectureMaster Node (“Control Plane”)

Etcd- Stores the state of the cluster- Doesn’t necessarily have to be co-located with other components- Must be backed up in a production scenario

kubelet- Agent for running Pods- Mounts volumes for Pods where required- Reports the status of Pods back to rest of system

kube-proxy- Enforces network rules on each Node (uses iptables)- Responsible for forwarding packets to correct destination

ArchitectureWorker Node

Master Node (api-server)- Takes an argument for etcd servers

Master Node (controller-manager)- Takes an argument for api server- Creates/defines virtual networks for containers and services- Takes an argument for cluster node CIDR- Takes an argument for service CIDR

kubelet- Configures the Docker bridge- Takes an address for the cluster DNS

kube-proxy- Takes an argument for the cluster node CIDR

ArchitectureNetworking

Various service components:- IaaS: EC2 / VPC- PaaS: Elastic Beanstalk / ECS- (No)SQL database services- Data Storage / Warehousing / Processing- Mobile Services- Serverless Services- CDN

AWSCloud Computing Platform

We will use CloudFormation to:- Launch EC2 instances into an existing VPC- Create a subnet for each kubernetes cluster- Create a route table for each subnet- Create Security Groups (firewall rules) for each cluster- Create Autoscale Groups for Master and Worker nodes

AWSToday: EC2, VPC and CloudFormation

Instance Configuration:- Userdata: Instructions to be run by AWS cloud-init system after boot- Chef: Userdata will instruct instances to bootstrap to Chef server

CloudFormation:- Method of keeping Infrastructure as Code- JSON based template that defines AWS Resources

AWSOther ways to build

Getting Started guide: http://kubernetes.io/docs/getting-started-guides/aws/ - $ set=something ; wget something | bash- Great for getting a cluster up and running quickly- Inflexible for integration into existing VPCs- Fussy if you put anything else in the VPC it creates

Kops: https://github.com/kubernetes/kops - “kubectl for clusters”- Will become the standard way to launch onto AWS- Still in alpha

Run with your own: https://github.com/kelseyhightower/kubernetes-the-hard-way - Takes some time- Expect to reverse-engineer- You will know exactly how the cluster is put together

Using the --cloud-provider=aws flag, the kubernetes components can be instructed to leverage AWS IaaS features.

Master instances (running controller-manager) must have an appropriate IAM role assigned.

Kubernetes can then- Create and destroy Elastic Load Balancers (ELBs)- Add and delete routes from cluster Route Table- Add and delete firewall rules on cluster Security Group

AWS and KubernetesKubernetes is able to configure AWS

Relevant resources must be appropriately tagged:- Name: KubernetesCluster- Value: ClusterId

AWS and KubernetesOur Workshop Architecture: Network

AWS and KubernetesOur Workshop Architecture: Servers

Hands-On

Build a cluster

● Choose yourself an ID for the cluster

$ git clone https://github.com/ContainerSolutions/kubernetes-aws-workshop.git$ cd kubernetes-aws-workshop/

$ ./build [user-id]

Configure kubectl

$ eval `ssh-agent`$ ssh-add /path/to/private.key

$ ./find-master [user-id]x.x.x.x$ ./set-cluster x.x.x.x

$ kubectl config view

Check the cluster status

$ kubectl cluster-info

$ kubectl get cs (componentstatus)

$ kubectl get nodes

$ kubectl get events

$ kubectl describe nodes

Deploy a container

$ kubectl create -f kube-files/nginx-pod.yml

$ kubectl get pods

$ kubectl describe pod nginx# note the pod ip address

Create a service

$ kubectl create -f kube-files/nginx-service.yml

$ kubectl get svc

$ kubectl describe service nginx-service# note the Endpoints# note the IP# note the NodePort

Investigate the service

$ kubectl describe service nginx-service

Name: nginx-serviceNamespace: defaultLabels: <none>Selector: app=nginxType: NodePortIP: 10.20.32.218Port: http 80/TCPNodePort: http 31975/TCPEndpoints: 10.100.0.2:80Session Affinity: None

$ ./run-nodes [user-id] curl -s [IP]$ ./run-nodes [user-id] curl -s [Endpoints]$ ./run-nodes [user-id] curl -s 127.0.0.1:[NodePort]

What’s happening?

$ ./find-nodes [cluster-id]x.x.x.xx.x.x.x

$ ssh ubuntu@x.x.x.x$ ip route list$ route -n$ sudo iptables -L -t nat

# view route table in AWS, note that the pod CIDRs are routed directly to an EC2 NIC

Cluster Add-Ons

$ kubectl create -f kube-files/kubernetes-dashboard.yml

$ kubectl proxyStarting to serve on 127.0.0.1:8001

# Go to 127.0.0.1:8001/ui

Cluster Add-Ons

$ kubectl create -f kube-files/kube-dns.yml

$ kubectl config use-context system

$ kubectl get pods# Note the pods you’ve not seen yet. These are running cluster services

$ kubectl config use-context workshop

Observing DNS

$ kubectl create -f kube-files/busybox.yml

$ kubectl exec -ti busybox sh

# nslookup google.com# nslookup nginx-service# nslookup kubernetes-dashboard.kube-system# cat /etc/resolv.conf# exit

Deploying a service

$ kubectl delete pod nginx$ kubectl delete svc nginx-service

$ kubectl create -f kube-files/nginx-deployment$ kubectl get pods$ kubectl get rs (replicaset)$ kubectl delete pod [nginx-pod]$ kubectl get pods

Deploying a service

$ kubectl expose deployment nginx --type=LoadBalancer

$ kubectl get svc -o wide# ...wait

Deploying a microservice application

$ kubectl create -f kube-files/microservices-demo.yml

$ kubectl get svc -o wide# ...wait

Tidy up...

$ kubectl delete service nginx$ kubectl delete deployment nginx

$ kubectl delete -f kube-files/microservices-demo.yml

$ ./delete [user-id]

$ ssh-agent -k

Questions?grant.ellis@container-solutions.com

kubernetes on aws

Software

media streaming on aws

rancherで簡単に作るk8s環境 kubernetes meetup tokyo...

aws black belt online seminar 2017 docker on aws

from dev to prod: kubernetes on aws (short ver.)

aerospike on aws

aws innovate: smart deployment on aws - andy kim

kubernetes on gcp

[jun aws 101] running lean on aws

docker on aws

serverless machine learning on aws - aws rome meetup

[오픈소스컨설팅]파일럿진행예제 on aws

moving to kubernetes on amazon web services at scale

chatting server on aws

scaling mongodb on aws

aws innovate: infrastructure automation on aws - seungdo...

drupal on vagrant-aws

security on aws - resources.trendmicro.com iam amazon...

ドカドカdocker on aws

oracle on aws 2014

serverless on kubernetes