拍云 ci/cd · • job/build：最的构建任务单元 •...

⼜又拍云 CI/CD 实践

莫红波-⼜又拍云

⾃自我介绍

2014 年年加⼊入⼜又拍云

图⽚片处理理业务

⾳音视频处理理业务

私有云项⽬目

边缘计算项⽬目

⾃自动化测试部署相关

主要内容

• GitLab WorkFlow

• ⼀一些术语

• 如何使⽤用 GitLab CI

• GitLab Runner 内部实现

• ⼀一些实践经验

GitLab WorkFlow

⼀一些术语

• Job/Build：最⼩小的构建任务单元• Stage：阶段，可以通过定义不不同的阶段来决定 Job 执⾏行行的先后顺序• Pipeline：流⽔水线，可以直观的看到不不同 Job 以及 Stage 间的关系• GitLabRunner：实际处理理 GitLab 构建任务的 Worker• GitLabMultiRunner：多个 GitlabRunner 的管理理者，⽀支持动态修改 runner 配置等• Executor：执⾏行行器器类型，每个 runner 都需要指定⼀一个，决定使⽤用哪个执⾏行行器器处理理构建任务

GitLab CI 使⽤用

GitLab CI 使⽤用 - runner 注册

GitLab CI 使⽤用 - 增加 .gitlab-ci.ymlstages: - test - buildtest: stage: test script: - echo skip tags: - opentalk

build: stage: build script: - export PATH=$PATH:/usr/local/go/bin - make docker VER=$CI_BUILD_TAG tags: - on-line-docker-builder only: - tags@consumers/jigsaw

GitLab CI 介绍

GitLab CI 介绍

115.231.11.11:80

GitLabMultiRunner

10.0.5.112

…

GitLabMultiRunner

10.0.5.113

…

GitLabMultiRunner

10.0.5.114

…

GitLab CI 内部消息机制 - runner 注册

POST /api/v4/runners HTTP/1.1Host: gitlab.widget-inc.comContent-Type: application/json

{ "info": { "name": "gitlab-ci-multi-runner", "version": "9.5.0", "revision": "413da38", "platform": "linux", "architecture": "amd64", "features": { "variables": false, "image": false, "services": false, "features": false, "cache": false } }, "token": "LhDVkUM9kysPbasGuEEd", "description": "runner for OpenTalk", "tag_list": "opentalk", "run_untagged": false, "locked": true}

请求体内容返回体内容

HTTP/1.1 201 CreatedContent-Type: application/json

{"id":136,"token":"5d36b61b35fbbfa432d8bce605c046"}

GitLab CI 内部消息机制 - 请求 Job

POST /api/v4/jobs/request HTTP/1.1Host: gitlab.widget-inc.comContent-Type: application/json

{ "info": { "name": "gitlab-ci-multi-runner", "version": "9.5.0", "revision": "413da38", "platform": "linux", "architecture": "amd64", "executor": "docker", "features": { "variables": true, "image": true, "services": true, "features": false, "cache": false } }, "token": "5d36b61b35fbbfa432d8bce605c046", "last_update": "b03cb53e63daeb53c7f7451c10c5cd9e"}

HTTP/1.1 204 No ContentX-Gitlab-Last-Update: b03cb53e63daeb53c7f7451c10c5cd9e

情况1: 没有需要处理理的 Job请求体内容




情况2: 有需要处理理的 Job请求体内容

HTTP/1.1 201 CreatedContent-Type: application/json

{ "id": 43848, "token": "8XdyE6KdnyUkDdZZQbxs", "allow_git_fetch": true, "job_info": { "name": "test", "stage": "test", "project_id": 2220, "project_name": "jigsaw" }, "git_info": { "repo_url": "http://gitlab-ci-token:[email protected]/consumers/jigsaw.git", "ref": "master", "sha": "bead0bb1a1d717bdd8eeaf1fe75905fdf4658781", "before_sha": "bead0bb1a1d717bdd8eeaf1fe75905fdf4658781", "ref_type": "branch" },




情况2: 有需要处理理的 Job 请求体内容

"runner_info": { "timeout": 3600 }, "variables": [{ "key": "CI_JOB_ID", "value": "43848", "public": true }, { "key": "CI_JOB_NAME", "value": "test", "public": true }], "steps": [{ "name": "script", "script": ["echo skip"], "timeout": 3600, "when": "on_success", "allow_failure": false }], "image": null, "services": [], "artifacts": [null], "cache": [null], "credentials": [], "dependencies": []}

GitLab CI 内部消息机制 - 确认 Job

PUT /api/v4/jobs/43848 HTTP/1.1Host: gitlab.widget-inc.comContent-Type: application/json

{ "info": { "name": "gitlab-ci-multi-runner", "version": "9.5.0", "revision": "413da38", "platform": "linux", "architecture": "amd64", "executor": "docker", "features": { "variables": true, "image": true, "services": true, "features": false, "cache": false } }, "token": "8XdyE6KdnyUkDdZZQbxs", "state": "running"}

返回体内容请求体内容

HTTP/1.1 200 OKContent-Type: application/json

null

GitLab CI 内部消息机制 - 信息返回

PATCH /api/v4/jobs/43848/trace HTTP/1.1Host: gitlab.widget-inc.comContent-Range: 0-354Content-Type: text/plainJob-Token: 8XdyE6KdnyUkDdZZQbxs

0KRunning with gitlab-ci-multi-runner 9.5.0 (413da38) on runner for OpenTalk (5d36b61b)Using Docker executor with image repo.upyun.com:5043/opentalk:latest ...ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?Will be retried in 3s ...

返回体内容

请求体内容

HTTP/1.1 202 AcceptedContent-Type: application/jsonJob-Status: running

"0-354"

GitLab CI 内部消息机制 - 结果返回

PUT /api/v4/jobs/43848 HTTP/1.1Host: gitlab.widget-inc.comContent-Type: application/json

{ "info": { "name": "gitlab-ci-multi-runner", "version": "9.5.0", "revision": "413da38", "platform": "linux", "architecture": "amd64", "executor": "docker", "features": { "variables": true, "image": true, "services": true, "features": false, "cache": false } }, "token": "8XdyE6KdnyUkDdZZQbxs", "state": "failed", "trace": "\u001b[0KRunning with gitlab-ci-multi-runner 9.5.0 (413da38)\n on runner for OpenTalk ..."}

返回体内容请求体内容

HTTP/1.1 200 OKContent-Type: application/json

true

GitLab CI 内部消息机制 - 总结

POST /api/v4/runners

201 Created

POST /api/v4/jobs/request

204 No Content

Runner

Job

GitLab CI 内部消息机制 - 总结

POST /api/v4/jobs/request

201 Created

PUT /api/v4/jobs/43848

200 OK

PATCH /api/v4/jobs/43848/trace

202 Accepted

PUT /api/v4/jobs/43848

200 OK

Job

GitLab Executor"job_info": {},"git_info": {},"runner_info": { "timeout": 3600 }, "variables": [{ "key": "CI_JOB_ID", "value": "43848", "public": true }, { "key": "CI_JOB_NAME", "value": "test", "public": true }], "steps": [{ "name": "script", "script": ["echo skip"], "timeout": 3600, "when": "on_success", "allow_failure": false }], "image": null, "services": [], "artifacts": [null], "cache": [null], "credentials": [], "dependencies": []}

GitLab Executor"job_info": {},"git_info": {},"runner_info": { "timeout": 3600 }, "variables": [{ "key": "CI_JOB_ID", "value": "43848", "public": true }, { "key": "CI_JOB_NAME", "value": "test", "public": true }], "steps": [{ "name": "script", "script": ["echo skip"], "timeout": 3600, "when": "on_success", "allow_failure": false }], "image": null, "services": [], "artifacts": [null], "cache": [null], "credentials": [], "dependencies": []}

Runner 是如何处理理 Job 的？内部处理理逻辑是怎样的？

GitLab Executor

GitLab Executor - BuildStagePrepare

Prepare

GitLab Executor - BuildStageGetSource

Prepare

GetSource

"git_info": { "repo_url": "http://gitlab-ci-token:[email protected]/consumers/jigsaw.git", "ref": "master", "sha": "bead0bb1a1d717bdd8eeaf1fe75905fdf4658781", "before_sha": "bead0bb1a1d717bdd8eeaf1fe75905fdf4658781", "ref_type": "branch" },

GitLab Executor - BuildStageRestoreCache

Prepare

GetSource

RestoreCache

"cache": [null],

GitLab Executor - BuildStageDownloadArtifacts

Prepare

GetSource

RestoreCache

DownloadArtifacts

"dependencies": { "id": 44231, "name": "compile-alpine", "token": "rpYZNZBbWekCdpJw4RFw", "artifacts_file": { "filename": "artifacts.zip", "size": 1810651 }}

GitLab Executor - BuildStageDownloadArtifacts

Prepare

GetSource

RestoreCache

DownloadArtifacts

"dependencies": { "id": 44231, "name": "compile-alpine", "token": "rpYZNZBbWekCdpJw4RFw", "artifacts_file": { "filename": "artifacts.zip", "size": 1810651 }}

GET /api/v4/jobs/44231/artifacts HTTP/1.1Host: gitlab.widget-inc.comJob-Token: rpYZNZBbWekCdpJw4RFwAccept-Encoding: gzip

GitLab Executor - BuildStageUserScripts

Prepare

GetSource

RestoreCache

DownloadArtifacts

ExecUserScripts

"steps": [{ "name": "script", "script": ["echo skip"], "timeout": 3600, "when": "on_success", "allow_failure": false }],

Prepare

GetSource

RestoreCache

DownloadArtifacts

ExecUserScripts

ExecAfterScripts

GitLab Executor - BuildStageAfterScripts

"steps": [{ "name": “after_script", "script": ["echo skip"], "timeout": 3600, "when": "on_success", "allow_failure": false }],

GitLab Executor - BuildStageArchiveCache

Prepare

GetSource

RestoreCache

DownloadArtifacts

ExecUserScripts

ExecAfterScripts

ArchiveCache

GitLab Executor - BuildStageUploadArtifacts

Prepare

GetSource

RestoreCache

DownloadArtifacts

ExecUserScripts

ExecAfterScripts

ArchiveCache

UploadArtifacts

"artifacts": [null],

GitLab Executor

Shell SSH Docker

Docker-SSH DockerMachine DockerMachine-SSH

VirtualBox Parallels Kubernetes

GitLab Executor

Shell SSH Docker

Docker-SSH DockerMachine DockerMachine-SSH

VirtualBox Parallels Kubernetes

•秒级别启动容器器

•轻量量

•资源隔离性好

•资源回收⽅方便便

Docker Executor

test: image: ubuntu:17.04 services: - mysql - redis script: - make && make start && make test

image: 本次测试环境使⽤用的 docker 镜像

services: 本次测试依赖的服务

script: 测试脚本

Docker Executor - Prepare

Services

Redis Mysql

Volumes

/build/proj /cache

Services

Redis Mysql

Volumes

/build/proj /cache

Predefine

VOLU

ME-FR

OM

Docker Executor - GetSource

Docker Executor - ExecScripts

Services

Redis Mysql

Build

Volumes

/build/proj /cache

VOLUME-FROM LINK

Docker Executor vs Docker SSH

$ docker run --name build --link=service-mysql:mysql —link=service-redis:redis —volumes-from=cache —volumes-from=source repo.upyun.com/shared /bin/bash < build_script.sh

docker

http://repo.upyun.com/shared

Docker Executor vs Docker SSH

$ docker run --name build --link=service-mysql:mysql —link=service-redis:redis —volumes-from=cache —volumes-from=source repo.upyun.com/shared /bin/bash < build_script.sh

$ docker run --name build --link=service-mysql:mysql —link=service-redis:redis —volumes-from=cache —volumes-from=source repo.upyun.com/shared-sshd

$ ssh root@container 'bash -s' < build_script.sh

docker

docker-ssh

http://repo.upyun.com/sharedhttp://repo.upyun.com/shared

GitLab Artifacts

MakeNamiImage

GitLab Artifacts - 优化前

MakeNamiImageCompileNamiBinary

GitLab Artifacts - 优化后

GitLab Artifacts - 优化后

stages: - compile - test - release

compile: stage: compile image: gcc:4.1.5 script: - make deps - make && make install && make tar tags: - compile artifacts: name: "binary" paths: - nami.tar.gz expire_in: "1 hour"

test: stage: test script: - make start - time make test tags: - offline-test

release: stage: release script: - make docker tags: - builder only: - tags@upyun-core/nami

GitLab Artifacts - 下载

GitLab Artifacts vs Cache

Cache 是为了了解决同个项⽬目同个 runner，前⼀一次 Job 与后⼀一次 Job 之间的缓存问题，所有数据都是缓存在 runner 所在的宿主机上

Artifacts 是为了了解决同个项⽬目不不同 runner不不同阶段之间的数据流通。所有数据都会保存在 GitLab 服务器器上，由于上传下载会影响测试效率

C/C++ 编译优化

CDN 项⽬目依赖 Nginx，存在公司定制逻辑，每次测试都需要重新编译 Nginx。

C/C++ 编译优化 - ccache

ccache（ “compiler cache” 的缩写）是⼀一个编译器器驱动器器。第⼀一次编译时 ccache 会缓存GCC 的 “-E” 输出、编译选项以及 .o ⽂文件到 $HOME/.ccache。第⼆二次编译时尽量量利利⽤用缓存，必要时更更新缓存。



测试跑完之后容器器⽴立刻就被回收了了，如何将编译缓存保留留到下⼀一次测试呢？



测试跑完之后容器器⽴立刻就被回收了了，如何将编译缓存保留留到下⼀一次测试呢？

$ ccache —set-config=cache_dir=/cache/.ccache$ ccache -F 0 && ccache -M 0

解决 Localhost 问题

Services

Redis Mysql

Build

LINK

解决 Localhost 问题

Services

Redis Mysql

Build

LINK

配置中更更多使⽤用 Localhost，⽅方便便本地测试

解决 Localhost 问题 - services 配置

Services

Redis Mysql

Build

LINK

解决 Localhost 问题 - services 配置

Services

Redis Mysql

Build

LINK

修改代码，使⽤用别

名访问

解决 Localhost 问题 - 端⼝口转发

before_script: - echo "tcppm 6379 redis 6379" > /tmp/3proxy.cfg - echo "tcppm 3306 mysql 3306" >> /tmp/3proxy.cfg - 3proxy /tmp/3proxy.cfg &

THANKS!

感谢您的聆听

拍云 ci/cd · • job/build：最的构建任务单元 •...

Documents