continuous integration & continuous delivery with gcp
TRANSCRIPT
Continuous Integration & Continuous Delivery with GCP
https://www.facebook.com/groups/GCPUG.TW/
https://plus.google.com/u/0/communities/116100913832589966421
Google Cloud Platform User Group Taiwan我們是Google Cloud Platform Taiwan User Group。在Google雲端服務在台灣地區展露頭角之後，
有許多新的服務、新的知識、新的創意，歡迎大家一起分享，一起了解 Google雲端服務...
GCPUG透過網際網路串聯喜好Google Cloud的使用者，分享與交流使用 GCP的點滴鑑驗。如果您
是Google Cloud Platform的初學者，您應該來聽聽前輩們的使用經驗；如果您是 Google Cloud Platform的Expert，您應該來分享一下寶貴的經驗，並與更多高手互相交流；如果您還沒開始用
Google Cloud Platform，那麼您應該馬上來聽聽我們是怎麼使用 Google Cloud的！
Hello!I am Cage Chung
I am here because I like to share my experiences.
You can find me at:http://kaichu.io
Outline
◎ App Engine Development
◎ Devops: CI & CD (Docker, gitlab, GCP)
◎ Tips & Study information
1.App Engine DevelopmentLet’s start with the first set of slides
Who is using GAE?
Angry Birds
Khan Academy
Royal Wedding
Snapchat
EA Games
Forbes
Bestbuy
https://cloud.google.com/customers/ Appengine
Google Cloud Platform | Compute
PaasIaaS
ApplicationDataRuntimeMiddlewareOSVirtualizationServersStorageNetworking
ApplicationDataRuntimeMiddlewareOSVirtualizationServersStorageNetworking
Saas
ApplicationDataRuntimeMiddlewareOSVirtualizationServersStorageNetworking
Infrastructure as a Service Platform as a Service Service as a Service
You manage Google manage
GAE Architecture
image: https://cloud.google.com/solutions/architecture/webapp
App Engine APIs
◎ Storage◎ Cloud Storage◎ Cloud SQL◎ Cloud Datastore◎ Memcache◎ Task Queue◎ Cloud Endpoints
Find more https://cloud.google.com/appengine/
App Engine runtime
Python2.7 Java7 Php Go
Ruby Node.js
standard
Flexible Beta
Python2.7/3.4 Java8 Go custom
[Google App Engine Documentation - App Engine — Google Cloud Platform](https://cloud.google.com/appengine/docs)
App Engine | waldo
queue
MetadataWaldo Server
App Engine
Task Queues
Cloud Storage
Optimal-waypoints
managed VMs(python27)
...
endpoints API(OAuth 2)
Object Notific
ation
CloudeDatastore
snapshot
managed VMs(nodejs)
Android App
CSV
upload waypoints
Store/retrieve metadata
waypoint map
snapshot
optimal road trip
RESTful APIs for waypoints list, upload
waypoints
standard
Flexible Beta
[Waldo-gcp](http://www.slideshare.net/cagechung/waldogcp)
2.Devops: CI & CD (Docker, gitlab, GCP)Let’s start with the second set of slides
Devops
DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.[1][2] It aims at establishing a culture and environment where building, testing, and releasing software, can happen rapidly, frequently, and more reliably - DevOps - Wikipedia
[image](https://goo.gl/pMC6UZ)
Origin imgae from and modified [DevOps：持續整合＆持續交付（Docker、CircleCI、AWS）](http://blog.amowu.com/2015/04/devops-continuous-integration-delivery-docker-circleci-aws-beanstalk.html)
Simple flow
◎ Python (gae-flask) ○ Local Development○ Local Test
◎ Gitlab◎ Code Review with GitHub(Gitlab) Flow◎ gitlab-ci-multi-runner
○ Local runner exec test docker○ Gitlab runner test docker
◎ Google App Engine○ Google Cloud Source Repositories○ Stackdriver Debugger
Local Development / Test | gae-flask
// clone repo
$ git clone [email protected]:cage1016/gae-flask.git
// make a new virtualenv
$ mkvirtualenv gae-flask
// pip install requirements.test.txt or link env as lib folder
(gae-flask-gitlab-ci) $ pip install -r requirements.testing.txt -t lib
// or
(gae-flask-gitlab-ci) $ pip install git+https://github.com/ze-phyr-us/linkenv.git
(gae-flask-gitlab-ci) $ linkenv env/lib/python2.7/site-packages lib
// local run app engine project
(gae-flask-gitlab-ci) $ dev_appserver.py app.yaml
// local run test
(gae-flask-gitlab-ci) $ python ./runtests.py
Local Development / mian.py
import flask
import requests
app = flask.Flask(__name__)
@app.route("/")
def hello():
return "hello world"
if __name__ == "__main__":
app.run()
Local Development / test_main.py
import pytest
import main
@pytest.fixture
def app():
return main.app
@pytest.fixture
def test_client(app):
return app.test_client()
def test_hello(test_client):
response = test_client.get("/")
assert response.data.decode("utf-8") == "hello world"
Gitlab Project Runner
https://gitlab.com/cage1016/gae-flask/runners
Gitlab Project Variables
gitlab-ci.yml
image: cage1016/gitlab-ci-gcp:v1.5
before_script:
- export CLOUDSDK_CORE_DISABLE_PROMPTS=1
- export CLOUDSDK_PYTHON_SITEPACKAGES=1
- export GCP_PROJECT=gae-lab-001
- export GCP_PROJECT_VERSION=uat
types:
- test
- deploy
test:
stage: test
script:
- sh ./scripts/tests.sh
deploy:
stage: deploy
script:
- sh ./scripts/deploy.sh
only:
master[GitLab Documentation](http://doc.gitlab.com/ee/ci/yaml/README.html#only-and-except)
gitlab-ci.yml (tests.sh)
#!/usr/bin/env bash
virtualenv env
source env/bin/activate
# install test env packages from requirements.testing.txt
pip install -r requirements.testing.txt -t lib
# link env as lib folder if you just execute `pip install -r requirements.testing.txt`
# pip install git+https://github.com/ze-phyr-us/linkenv.git
# linkenv env/lib/python2.7/site-packages lib
# run tests
python runtests.py
gitlab-ci.yml (deploy.sh)
#!/usr/bin/env bash
pip install -r requirements.txt -t lib/
echo $GCLOUD_KEY > key.json
gcloud auth activate-service-account $GCLOUD_ACCOUNT --key-file key.json
gcloud --quiet config set project $GCP_PROJECT
gcloud --quiet preview app deploy app.yaml --no-promote --version $GCP_PROJECT_VERSION
Local runner exec test docker
// [install gitlab-ci-multi-runner](https://goo.gl/a99Aa4)
// list and register gitlab-ci-runner
$ gitlab-ci-multi-runner list
Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml
gae-flask-gitlab-ci Executor=docker Token=95d56b26104856b9f3852f2d82b1e5 URL=https://gitlab.com/ci
// prepare a docker-machine for gitlab-ci-multi-runner
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0
// run a gitlab-ci-multi-runner test via docker
$ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY
GCLOUD_ACCOUNT=$GCLOUD_ACCOUNT GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION"
// push git
(gae-flask-gitlab-ci) $ git push master
Docker cage1016/gitlab-ci-gcp:v1.5
FROM buildpack-deps:jessie
# remove several traces of debian python
RUN apt-get purge -y python.*
RUN apt-get update && apt-get install -y -qq --no-install-recommends unzip openssh-client python-openssl
# install "virtualenv", since the vast majority of users of this image will want it
RUN pip install --no-cache-dir virtualenv
# Install the Google Cloud SDK.
# archive gcloud sdk https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-105.0.0-
linux-x86_64.tar.gz
ENV HOME /
ENV CLOUDSDK_PYTHON_SITEPACKAGES 1
RUN wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-102.0.0-linux-x86_64.
tar.gz && tar -zxvf google-cloud-sdk-102.0.0-linux-x86_64.tar.gz && rm google-cloud-sdk-102.0.0-linux-
x86_64.tar.gz
RUN google-cloud-sdk/install.sh --usage-reporting=true --path-update=true --bash-completion=true --rc-
path=/.bashrc --additional-components app-engine-java app-engine-python app kubectl alpha beta
...
https://github.com/cage1016/gitlab-ci-gcp/blob/master/Dockerfile
Gitlab Build Status
https://gitlab.com/cage1016/gae-flask/builds
Gitlab Build detail
https://gitlab.com/cage1016/gae-flask/builds/1321684
Push a new branch to Gitlab
https://gitlab.com/cage1016/gae-flask/builds
Google Cloud Source Repositories repo
$ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default
// generates a file named source-context.json, which contains information about the version of the
source code used to build the application
$ gcloud preview app gen-repo-info-file
// push code to google cloud source repositories
$ git push google master[/ - GAE Labs](https://goo.gl/c99Dz8)
3.Tips & Study informationLet’s start with the third set of slides
“5G Google cloud storage for App
engine for FREE
<project-id>.appspot.comstaging.<project-id>.appspot.com
“App Engine Managed VMs is
renamed to App Engine flexible environment. March 24, 2016
[Python Release Notes - Python — Google Cloud Platform](https://cloud.google.com/appengine/docs/python/release-notes#1.9.35)
“App Engine Admin API
Cloud Source Repositories API Google Cloud Storage
“At the moment, the Go app engine SDK is still based on Go 1.4, even if you use the go1.6beta version for
your app. This means that the SDK doesn't understand 1.6-style vendoring, and so doesn't correctly upload the vendored dependencies.
This will be fixed when 1.6 becomes the stable (go1) version and is used as the basis of the SDK. Stay
tuned!
[gcpug.tw.go/.travis.yml at master · gcpug-tw/gcpug.tw.go](https://github.com/gcpug-tw/gcpug.tw.go/blob/master/.travis.yml)[Vendorization for deployment - Google Groups](https://groups.google.com/forum/#!topic/google-appengine-go/umcPV_4tAm8)
Study information
◎ Gitlab○ [Quick Start](http://doc.gitlab.com/ce/ci/quick_start/README.
html)○ [Configuration of your builds with .gitlab-ci.yml](http://doc.gitlab.
com/ee/ci/yaml/README.html)○ [GitLab.org / gitlab-ci-multi-runner · GitLab](https://gitlab.
com/gitlab-org/gitlab-ci-multi-runner)◎ Deploying to App Engine or Managed VMs from Travis CI
○ [Deploying to App Engine or Managed VMs from Travis CI - YouTube](https://www.youtube.com/watch?v=7U4jjRw_AJk&feature=youtu.be)
○ [GoogleCloudPlatform/continuous-deployment-demo](https://github.com/GoogleCloudPlatform/continuous-deployment-demo)
Study information cont.
◎ GCP○ [Cloud Source Repositories - Private Git Repositories — Google
Cloud Platform](https://cloud.google.com/source-repositories/)○ [Stackdriver Debugger - Production Debugging - Google Cloud
Platform — Google Cloud Platform](https://cloud.google.com/debugger/)
○ [Managed VMs lab · kaichu.io](http://kaichu.io/2015/09/managed-vms-lab/)
