Download - Kubernetes on AWS
Pre-reqs:● Git: https://git-scm.com/book/en/v2/Getting-Started-Installing-Git
● AWS CLI: http://docs.aws.amazon.com/cli/latest/userguide/installing.html
● kubectl:http://cs-k8s-workshop.s3.amazonaws.com/kubectl/darwin/amd64/kubectl
http://cs-k8s-workshop.s3.amazonaws.com/kubectl/linux/amd64/kubectl
● Bash
● git clone https://github.com/ContainerSolutions/kubernetes-aws-workshop
www.container-solutions.com | [email protected]
Kubernetes on AWSGrant [email protected]
www.container-solutions.com | [email protected]
Who’s who
● Presenters
● You!
➔ Developers? Ops? DevOps?
➔ Tools, languages & frameworks?
➔ Familiar or using any orchestration platform? Mesos/Swarm/ECS?
www.container-solutions.com | [email protected]
Purpose of the Workshop
● Get an overview of the components in kubernetes
● See how kubernetes leverages features present in AWS
● Get an idea of how a production setup may take shape
www.container-solutions.com | [email protected]
Scope of the Workshop
● Basic features of Kubernetes
● Brief look at AWS CloudFormation and IaaS components
● Hands on
www.container-solutions.com | [email protected]
Kubernetes
www.container-solutions.com | [email protected]
Kubernetes
● From the Greek meaning “Helmsman” or “Pilot”
● Founded by Joe Beda, Brendan Burns and Craig McLuckie
● First announced by Google in 2014
www.container-solutions.com | [email protected]
www.container-solutions.com | [email protected]
Basic concepts
● Pods● Labels / Selectors● Replication Controllers / Replica Sets● Deployments● Services
All Resources can be expressed as YAML or JSON files
www.container-solutions.com | [email protected]
Pods● A pod is one or more containers● Ensures co-location / shared fate● Pods are scheduled, then do not move between nodes● Containers share resources within the pod:
➔ Volumes➔ Network / IP➔ Port space➔ CPU / Memory allocations
www.container-solutions.com | [email protected]
Pod exampleapiVersion: v1kind: Podmetadata: labels: name: influxdb name: influxdbspec: containers: - image: docker.io/tutum/influxdb:latest name: influxdb ports: - containerPort: 8083 name: admin protocol: TCP - containerPort: 8086 name: http protocol: TCP
www.container-solutions.com | [email protected]
Labels / Selectors
● Labels are arbitrary metadata● Attachable to nearly all API objects
➔ e.g.: Pods, ReplicationControllers, Services...● Simple key=value pairs● Can be queried with selectors
www.container-solutions.com | [email protected]
Labels example
- release=stable, release=canary- environment=dev, environment=qa, environment=prod- tier=frontend, tier=backend, tier=middleware- partition=customerA, partition=customerB- etc…
www.container-solutions.com | [email protected]
Labels example
www.container-solutions.com | [email protected]
Selectors explained
Labels are queryable metadata - selectors can do the queries:- Equality based:
- environment = production- tier != frontend- combinations: tier != frontend, version = 1.0.0
- Set based:- environment in (production, pre-production)- tier notin (frontend, backend)- partition or !partition
www.container-solutions.com | [email protected]
Selectors example
www.container-solutions.com | [email protected]
Replication Controllers
● Define the number of replicas of a pod● Will scheduled across all applicable nodes● Can change replica value to scale up/down● Which pods are scaled depends on RC selector● Labels and selectors are used for grouping● Can do quite complex things with RCs and labels
www.container-solutions.com | [email protected]
Example Replication ControllerapiVersion: v1kind: ReplicationControllermetadata: name: nginxspec: replicas: 3 selector: app: nginx template: metadata: name: nginx labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80
www.container-solutions.com | [email protected]
Replica Set
Replica Set is the next-generation Replication Controller. The only difference between a Replica Set and a Replication Controller right now is the selector support. Replica Set supports the new set-based selector which allow filtering keys according to a set of values:
- In- Notin- exists (only the key identifier)
For example:environment in (production, qa)
tier notin (frontend, backend)
partition
!partition
www.container-solutions.com | [email protected]
Deployments
A Deployment is responsible for creating and updating instances of your application
● Create a Deployment to bring up Pods and a replica set.
● Check the status of a Deployment to see if it succeeds or not.
● Later, update that Deployment to recreate the Pods (for example, to use a new image).
● Rollback to an earlier Deployment revision if the current Deployment isn’t stable.
● Pause and resume a Deployment.
www.container-solutions.com | [email protected]
Deployment exampleapiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginx-deploymentspec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 minReadySeconds: 5 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.91 ports: - containerPort: 80
www.container-solutions.com | [email protected]
Services
“defines a logical set of Pods and a policy by which to access them”
● As Pods are ephemeral, we can't depend on Pod IPs
● Services find pods that match certain selection criteria
● Services can load balance between multiple Pods
● Services can have a single IP that doesn’t change
www.container-solutions.com | [email protected]
Services
A group of pods that act as one == Service- group == selector
Defines access policy- LoadBalanced, NodePort
Gets a stable virtual IP and Port- Called the service portal- Also a DNS name- On prem additional loadbalancer is needed
VIP is captured by kube-proxy- Watches the service consistency- Updates when backend changes
www.container-solutions.com | [email protected]
Service example
www.container-solutions.com | [email protected]
Service example
apiVersion: v1kind: Servicemetadata: name: railsappspec: type: NodePort selector: app: railsapp ports: - name: http nodePort: 36000 port: 80 protocol: TCP
www.container-solutions.com | [email protected]
Architecture
etcd (stores cluster state)API ServerSchedulerController managerKubelet (“node agent”)Kube-proxyContainer Runtime
https://github.com/kubernetes/kubernetes/blob/release-1.3/docs/design/architecture.md
www.container-solutions.com | [email protected]
ArchitectureMaster Node (“Control Plane”)
Api server- Point of interaction with the cluster- Exposes an http endpoint
Controller Manager- Responsible for most of the important stuff- Interacts with the api server to retrieve cluster state- Responsible for configuring networking- Allocates node CIDRs- Ensures correct number of pods are running- Reacts to Nodes being added / deleted- Manages Service Accounts and security tokens
Scheduler - Schedules newly created pods to a Node
www.container-solutions.com | [email protected]
ArchitectureMaster Node (“Control Plane”)
Etcd- Stores the state of the cluster- Doesn’t necessarily have to be co-located with other components- Must be backed up in a production scenario
www.container-solutions.com | [email protected]
kubelet- Agent for running Pods- Mounts volumes for Pods where required- Reports the status of Pods back to rest of system
kube-proxy- Enforces network rules on each Node (uses iptables)- Responsible for forwarding packets to correct destination
ArchitectureWorker Node
www.container-solutions.com | [email protected]
Master Node (api-server)- Takes an argument for etcd servers
Master Node (controller-manager)- Takes an argument for api server- Creates/defines virtual networks for containers and services- Takes an argument for cluster node CIDR- Takes an argument for service CIDR
kubelet- Configures the Docker bridge- Takes an address for the cluster DNS
kube-proxy- Takes an argument for the cluster node CIDR
ArchitectureNetworking
www.container-solutions.com | [email protected]
ArchitectureNetworking
www.container-solutions.com | [email protected]
AWS
www.container-solutions.com | [email protected]
Various service components:- IaaS: EC2 / VPC- PaaS: Elastic Beanstalk / ECS- (No)SQL database services- Data Storage / Warehousing / Processing- Mobile Services- Serverless Services- CDN
AWSCloud Computing Platform
www.container-solutions.com | [email protected]
We will use CloudFormation to:- Launch EC2 instances into an existing VPC- Create a subnet for each kubernetes cluster- Create a route table for each subnet- Create Security Groups (firewall rules) for each cluster- Create Autoscale Groups for Master and Worker nodes
AWSToday: EC2, VPC and CloudFormation
Instance Configuration:- Userdata: Instructions to be run by AWS cloud-init system after boot- Chef: Userdata will instruct instances to bootstrap to Chef server
CloudFormation:- Method of keeping Infrastructure as Code- JSON based template that defines AWS Resources
www.container-solutions.com | [email protected]
AWSOther ways to build
Getting Started guide: http://kubernetes.io/docs/getting-started-guides/aws/ - $ set=something ; wget something | bash- Great for getting a cluster up and running quickly- Inflexible for integration into existing VPCs- Fussy if you put anything else in the VPC it creates
Kops: https://github.com/kubernetes/kops - “kubectl for clusters”- Will become the standard way to launch onto AWS- Still in alpha
Run with your own: https://github.com/kelseyhightower/kubernetes-the-hard-way - Takes some time- Expect to reverse-engineer- You will know exactly how the cluster is put together
www.container-solutions.com | [email protected]
Using the --cloud-provider=aws flag, the kubernetes components can be instructed to leverage AWS IaaS features.
Master instances (running controller-manager) must have an appropriate IAM role assigned.
Kubernetes can then- Create and destroy Elastic Load Balancers (ELBs)- Add and delete routes from cluster Route Table- Add and delete firewall rules on cluster Security Group
AWS and KubernetesKubernetes is able to configure AWS
Relevant resources must be appropriately tagged:- Name: KubernetesCluster- Value: ClusterId
www.container-solutions.com | [email protected]
AWS and KubernetesOur Workshop Architecture: Network
www.container-solutions.com | [email protected]
AWS and KubernetesOur Workshop Architecture: Servers
www.container-solutions.com | [email protected]
Hands-On
www.container-solutions.com | [email protected]
Build a cluster
● Choose yourself an ID for the cluster
$ git clone https://github.com/ContainerSolutions/kubernetes-aws-workshop.git$ cd kubernetes-aws-workshop/
$ ./build [user-id]
www.container-solutions.com | [email protected]
Configure kubectl
$ eval `ssh-agent`$ ssh-add /path/to/private.key
$ ./find-master [user-id]x.x.x.x$ ./set-cluster x.x.x.x
$ kubectl config view
www.container-solutions.com | [email protected]
Check the cluster status
$ kubectl cluster-info
$ kubectl get cs (componentstatus)
$ kubectl get nodes
$ kubectl get events
$ kubectl describe nodes
www.container-solutions.com | [email protected]
Deploy a container
$ kubectl create -f kube-files/nginx-pod.yml
$ kubectl get pods
$ kubectl describe pod nginx# note the pod ip address
www.container-solutions.com | [email protected]
Create a service
$ kubectl create -f kube-files/nginx-service.yml
$ kubectl get svc
$ kubectl describe service nginx-service# note the Endpoints# note the IP# note the NodePort
www.container-solutions.com | [email protected]
Investigate the service
$ kubectl describe service nginx-service
Name: nginx-serviceNamespace: defaultLabels: <none>Selector: app=nginxType: NodePortIP: 10.20.32.218Port: http 80/TCPNodePort: http 31975/TCPEndpoints: 10.100.0.2:80Session Affinity: None
$ ./run-nodes [user-id] curl -s [IP]$ ./run-nodes [user-id] curl -s [Endpoints]$ ./run-nodes [user-id] curl -s 127.0.0.1:[NodePort]
www.container-solutions.com | [email protected]
What’s happening?
$ ./find-nodes [cluster-id]x.x.x.xx.x.x.x
$ ssh [email protected]$ ip route list$ route -n$ sudo iptables -L -t nat
# view route table in AWS, note that the pod CIDRs are routed directly to an EC2 NIC
www.container-solutions.com | [email protected]
Cluster Add-Ons
$ kubectl cluster-info
$ kubectl create -f kube-files/kubernetes-dashboard.yml
$ kubectl proxyStarting to serve on 127.0.0.1:8001
# Go to 127.0.0.1:8001/ui
www.container-solutions.com | [email protected]
Cluster Add-Ons
$ kubectl create -f kube-files/kube-dns.yml
$ kubectl config use-context system
$ kubectl get pods# Note the pods you’ve not seen yet. These are running cluster services
$ kubectl config use-context workshop
$ kubectl cluster-info
www.container-solutions.com | [email protected]
Observing DNS
$ kubectl create -f kube-files/busybox.yml
$ kubectl exec -ti busybox sh
# nslookup google.com# nslookup nginx-service# nslookup kubernetes-dashboard.kube-system# cat /etc/resolv.conf# exit
www.container-solutions.com | [email protected]
Deploying a service
$ kubectl delete pod nginx$ kubectl delete svc nginx-service
$ kubectl create -f kube-files/nginx-deployment$ kubectl get pods$ kubectl get rs (replicaset)$ kubectl delete pod [nginx-pod]$ kubectl get pods
www.container-solutions.com | [email protected]
Deploying a service
$ kubectl expose deployment nginx --type=LoadBalancer
$ kubectl get svc -o wide# ...wait
www.container-solutions.com | [email protected]
Deploying a microservice application
$ kubectl create -f kube-files/microservices-demo.yml
$ kubectl get svc -o wide# ...wait
www.container-solutions.com | [email protected]
Tidy up...
$ kubectl delete service nginx$ kubectl delete deployment nginx
$ kubectl delete -f kube-files/microservices-demo.yml
$ ./delete [user-id]
$ ssh-agent -k
www.container-solutions.com | [email protected]