60分鐘完送百萬EDM，背後雲端CI/CD實戰大公開

HELLO!I am Cage Chung

I am here because I like to share my experiences. You can find me at http://kaichu.io / QNAP 雲端應用部資深工程師

https://www.facebook.com/groups/GCPUG.TW/

https://plus.google.com/u/0/communities/116100913832589966421

[您知道”GCPUG”要怎麼唸嗎？為什麼會有一隻狗在 Logo裡面呢？ ]

Google Cloud Platform User Group的縮寫是GCPUGGCPUG直接唸成G.C.P.U.G？當然可以！

但它也可以分開來，唸成 G.C. PUG喔～

Pug，指的是巴哥犬，所以 GCPUG的Logo中間才會有一隻可愛的巴哥犬喲。

下次聽到別人說G.C. PUG 的時候，您就可以大聲說：「我也是G.C. PUG社團成員！」

17th Meetup

大綱

◉ EDM 二三事 & 送信服務架構◉ CI (持續整合) & CD (持續部署) 整合◉ Demo/Cloud Source Repositories, Stackdriver Debugger◉ 小提示 & 學習資源

EDM 二三事 & 送信服務架構Let’s start with the first set of slides 1

寄送 Email 好簡單 - API Call

# Google App Engine Send Email example (Python)

from google.appengine.api import app_identity

from google.appengine.api import mail

def send_approved_mail(sender_address):

# [START send_mail]

mail.send_mail(sender=sender_address,

to="Albert Johnson <Albert.Johnson@example.com>",

subject="Your account has been approved",

body="""Dear Albert:

Your example.com account has been approved. You can now visit

http://www.example.com/ and sign in using your Google Account to

access new features.

Please let us know if you have any questions.

The example.com Team

""")

# [END send_mail]

send_approved_mail('{}@appspot.gserviceaccount.com'.format(app_identity.get_application_id()))

客戶最關心的事 - 送達率

Sender(Bob)

Recipients(Alice / Sunrain)

Email Reputation

Sender Score Average Delivered

0-60 21%

61-70 67%

71-80 68%

81-90 92%

91-100 95%

[Your Email Reputation](http://www.slideshare.net/emaildelivered/email-reputation-2)[5 Ways to Check Your Sending Reputation | SendGrid](https://sendgrid.com/blog/5-ways-check-sending-reputation/)

◉ Authentication

◉ Bounce management

◉ List cleanliness

◉ User engagement (recipient

feedback)

EDM 實際會面臨到的種種問題...

Email Reputation

SpamminessWhiteList

Shared IP

DK(IM)/SPF

Blocks

Invalid Emails

Blacklists

SpamDedicated IP

Scores

[Best Practices in Email Deliverability - Act-On Software](https://www.act-on.com/ebook/email-marketing-deliverability-ebook/)

[垃圾邮件检测结果](http://www.mail-tester.com/web-eBR3ar&lang=zh)

Mail Tester

Sendgrid 如何幫我們送信

[Email Message Flow Explained - SendGrid | SendGrid](https://sendgrid.com/docs/Classroom/Basics/Email_Infrastructure/email_flow.html)

Sendgrid 遊戲規則

[IP Warmup Schedule - SendGrid Documentation | SendGrid](https://sendgrid.com/docs/API_Reference/Web_API_v3/IP_Management/ip_warmup_schedule.html)

[SendGrid | Heroku Dev Center](https://devcenter.heroku.com/articles/sendgrid)

送信服務架構

我們所使用到的資源

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

A fast, economical and fully managed data warehouse for

large-scale data analytics

BigQuery

Cloud Datastore is a highly-scalable NoSQL database for

your web and mobile applications.

DataStore

A powerful platform to build web and mobile apps that

scale automatically.

App Engine

A Powerful, Simple and Cost Effective Object Storage Service.

Cloud Storage

Delivering your transactional and marketing email through one reliable platform.

Sendgrid

Create RESTful APIs on App Engine accessible to iOS, Android and Javascript clients.

Cloud Endpoints

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

UI 彈性化擴展層、排程器、 Tasks 處理。針對不同任務的機器予給優化的資源，並可自動進行擴展

EDM Server

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

Cloud logging / sendgrid webooks (tracking - bounce, click, open, spam) 的資料值串接 BigQuery, 作後序分析

資料分析

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

檔案暫存

EDM html 檔案、連絡人清單暫存區

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

RESTful API endpoint

提供 RESTful API 給前端應用程式

https://<gcp-project-id>.appspot.com/_ah/api/explorerhttp://localhost:8080/_ah/api/explorer

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

Sendgrid

送信服務商，提供豐富的報表、EDM 相關的 know how，亦有 webhook 可以串接

App Engine

BigQuery

Clou

d St

orag

e

DataStore

Clou

d En

dpoi

nts

Sendgrid

Metadata 儲存

Nosql datastore. 儲存 EDM 送信服務過程中相關 Metadata. 連絡人清單、Log 等

架構

queue

MetadataEDM Server

App Engine

Task Queues

Cloud Storage

endpoints API(OAuth 2)

CloudeDatastore

CSV

upload recipients

Store/retrieve metadata

RESTful APIs for recipients

mail send

cron

...

# dispatch.yaml

application: your-project-id

dispatch: - url: "*/favicon.ico" module: default

- url: "your-project-id.appspot.com/" module: default

- url: "*/tasks/*" module: tasks

- url: "*/googleae8f4bcce8bec00c.html" module: ownership

- url: "*/*" module: "default"

# default/app.yaml

automatic_scaling:

min_idle_instances: 1

# tasks/app.yaml

instance_class: F4

automatic_scaling:

min_idle_instances: 2

min_pending_latency: 30ms

max_pending_latency: 300ms

# max_concurrent_requests: 80

# max_idle_instances: automatica # default value

# min_pending_latency: 30ms # default value

# max_pending_latency: automatic

# max_concurrent_requests: 50

EDM Server

EDM 檔案

聯絡人清單

Queue

API

Retry

fail

Retry cron

Schedule cron SuccessLog

流程示意圖 failLog

CI(持續整合) & CD (持續部署) 整合Let’s start with the second set of slides 2

Image by[每个架构师都应该研究下康威定律](http://www.infoq.com/cn/articles/every-architect-should-study-conway-law)

Devops: CI & CD (Docker, gitlab, GCP)

Bitbucket

Github

Jenkins

Travis CI

CircleCI

Docker

Gitlab

Cloud Source Repositories

Stackdriver Debugger

Origin imgae from and modified [DevOps：持續整合＆持續交付（Docker、CircleCI、AWS）](http://blog.amowu.com/2015/04/devops-continuous-integration-delivery-docker-circleci-aws-beanstalk.html)

簡單腳本

◉ Local Development / Test◉ Gitlab-ci-multi-runner

◉ Local runner exec test - docker◉ Gitlab runner test docker◉ gcloud deploy

◉ Google App Engine◉ Google Cloud Source Repositories◉ Stackdriver Debugger

Gitlab Project Runner

Gitlab Project Variables

Register Gitlab runner

$ gitlab-ci-multi-runner register

WARNING: Running in user-mode.

WARNING: The user-mode requires you to manually start builds processing:

WARNING: $ gitlab-runner run

WARNING: Use sudo for system-mode:

WARNING: $ sudo gitlab-runner...

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci):

https://gitlab.com/ci

Please enter the gitlab-ci token for this runner:

<your-gitlab-project-runner-register-token>

Please enter the gitlab-ci description for this runner:

[Kai-Chus-MBP]: edm

Please enter the gitlab-ci tags for this runner (comma separated):

gae,python,edm

Registering runner... succeeded runner=kvwzB1cE

Please enter the executor: ssh, shell, parallels, docker, docker-ssh, virtualbox:

docker

Please enter the default Docker image (eg. ruby:2.1):

cage1016/gitlab-ci-gcp:v1.6

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

.gitlab-ci.yamlimage: cage1016/gitlab-ci-gcp:v1.6

before_script:

- export CLOUDSDK_CORE_DISABLE_PROMPTS=1

- export CLOUDSDK_PYTHON_SITEPACKAGES=1

- export GCP_PROJECT=cage-20160705-edm

- export GCP_PROJECT_VERSION=uat

types:

- test

- deploy

test:

stage: test

script:

- sh ./scripts/tests.sh

deploy:

stage: deploy

script:

- sh ./scripts/deploy.sh

only:

- develop

.gitlab-ci.yaml (./scripts/tests.sh)

#!/usr/bin/env bash

virtualenv env

source env/bin/activate

# install Google App engine test env packages from requirements.testing.txt

pip install -r requirements.testing.txt

echo $GCLOUD_SERVICE_JSON_KEY > tasks/cage-20160705-edm-cc07ec5bf115.json

# run tests

py.test tasks/tests

.gitlab-ci.yaml (./scripts/deploy.sh)

#!/usr/bin/env bash

echo $GCLOUD_KEY > key.json

gcloud auth activate-service-account $GCLOUD_ACCOUNT --key-file key.json

gcloud --quiet config set project $GCP_PROJECT

# tasks module

pip install -r tasks/requirements.txt -t tasks/lib/

gcloud --quiet preview app deploy tasks/app.yaml --no-promote --version $GCP_PROJECT_VERSION --project

$GCP_PROJECT

# gen-repo-info-file

gcloud app gen-repo-info-file

git push google develop

Local runner exec test docker

// [install gitlab-ci-multi-runner](https://goo.gl/a99Aa4)

// list and register gitlab-ci-runner

$ gitlab-ci-multi-runner list

Listing configured runners ConfigFile=/Users/cage/.gitlab-runner/config.toml

edm Executor=docker Token=<your-token> URL=https://gitlab.com/ci

// prepare a docker-machine for gitlab-ci-multi-runner

$ docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS

runner - virtualbox Running tcp://192.168.99.100:2376 v1.11.0

// run a gitlab-ci-multi-runner test via docker

$ gitlab-ci-multi-runner --debug exec docker test --env "GCLOUD_KEY=$GCLOUD_KEY" --env " GCLOUD_ACCOUNT=

$GCLOUD_ACCOUNT" --env="GCP_PROJECT_VERSION=$GCP_PROJECT_VERSION"

$ git remote add google https://source.developers.google.com/p/[PROJECT_ID]/r/default

// generates a file named source-context.json, which contains information about the version of the source

code used to build the application

$ gcloud preview app gen-repo-info-file

// push code to google cloud source repositories

$ git push google master Google Cloud Source Repositories - List

Google Cloud Source Repositories - Source Code

Stackdriver Debugger | Source Code

Stackdriver Debugger | Debug

Image by [每个架构师都应该研究下康威定律](http://www.infoq.com/cn/articles/every-architect-should-study-conway-law)

Demo/Cloud Source Repositories, Stackdriver Debugger Let’s start with the third set of slides 3

小提示 & 學習資源Let’s start with the fourth set of slides 4

“5G Google cloud storage for App engine for FREE

<project-id>.appspot.comstaging.<project-id>.appspot.com

“App Engine Admin API

Cloud Source Repositories API Google Cloud Storage

學習資源

◉ [Best Practices in Email Deliverability - Act-On Software](https://www.act-on.com/ebook/email-marketing-deliverability-ebook/)

◉ [Getting Started - SendGrid Documentation | SendGrid](https://sendgrid.com/docs/index.html)

◉ [Newsletters spam test by mail-tester.com](http://www.mail-tester.com/)

◉ [Your Email Reputation](http://www.slideshare.net/emaildelivered/email-reputation-2)

學習資源

◉ Deploying to App Engine or Managed VMs from Travis CI◉ [Deploying to App Engine or Managed VMs from Travis CI -

YouTube](https://www.youtube.com/watch?v=7U4jjRw_AJk&feature=youtu.be)

◉ [GoogleCloudPlatform/continuous-deployment-demo](https://github.com/GoogleCloudPlatform/continuous-deployment-demo)

學習資源

◉ GCP◉ [Cloud Source Repositories - Private Git Repositories —

Google Cloud Platform](https://cloud.google.com/source-repositories/)

◉ [Stackdriver Debugger - Production Debugging - Google Cloud Platform — Google Cloud Platform](https://cloud.google.com/debugger/)

◉ [Managed VMs lab · kaichu.io](http://kaichu.io/2015/09/managed-vms-lab/)

DevOps (Development & Operations)

Install the necessary apps with one-click on demand and the system will automatically finish the set-up processQuickly build a development environment ideally suited for collaborationOne NAS with Multiple projects

Recommended apps for quick installation : Private Registry, GitLab, and Jenkins.

THANKS!Any questions?

You can find me athttp://kaichu.io / cage.chung@gmail.com