![Page 1: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/1.jpg)
![Page 2: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/2.jpg)
Implementation of ContinuousIntegration for Linux ImagesJerome Belleman, Linux Support Team
13 October 2017 Implementation of CI for Linux Images 2
![Page 3: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/3.jpg)
Context
13 October 2017 Implementation of CI for Linux Images 3
![Page 4: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/4.jpg)
Operating System Image Production
� Linux Support Team make images
� Built with Koji
� Those images you see when you run:
% openstack image list
+-----------------+--------------------------------+
| ID | Name |
+-----------------+--------------------------------+
| 29262d...3 c8fc9 | CC7 - x86_64 [2017 -04 -06] |
| d8c28e ... a9015b | SLC6 - x86_64 [2017 -04 -06] |
| f1c040 ...0 c92c4 | SLC6 - i686 [2017 -04 -06] |
| b2b7ba ... e5c9c3 | CC7 TEST - x86_64 [2017 -04 -06] |
| ... | |
13 October 2017 Implementation of CI for Linux Images 4
![Page 5: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/5.jpg)
It’s Critical That. . .
� We build images quickly
� We build images autonomously
� We make sure images work flawlessly
13 October 2017 Implementation of CI for Linux Images 5
![Page 6: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/6.jpg)
Testing Out Images
Perform some manual checks:
� Try starting a VM
� Try logging into it
� Check Puppet reports
� Check cloud-init logs
� A list of tests
13 October 2017 Implementation of CI for Linux Images 6
![Page 7: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/7.jpg)
We Need To. . .
� Automate test execution
� Perform more thorough tests
� Be able to manage tests
13 October 2017 Implementation of CI for Linux Images 7
![Page 8: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/8.jpg)
Towards a Variety of Tests
13 October 2017 Implementation of CI for Linux Images 8
![Page 9: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/9.jpg)
So Far. . .
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
ai-bs-vm
13 October 2017 Implementation of CI for Linux Images 9
![Page 10: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/10.jpg)
A Job to Build an Image
� Runs koji image-build
� Once a day – check if already built today
� Downloads the image from Koji
� Uploads the image into OpenStack
13 October 2017 Implementation of CI for Linux Images 10
![Page 11: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/11.jpg)
A Job to Create a Test VM
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
ai-bs-vm
13 October 2017 Implementation of CI for Linux Images 11
![Page 12: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/12.jpg)
A Puppet Test VM
Purpose:
� A prerequisite
� A test in its own right
Not much in cern/cc7-base → before_script:
� Copy .repo files
� Install ai-tools
What it does:
� Run ai-bs-vm
13 October 2017 Implementation of CI for Linux Images 12
![Page 13: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/13.jpg)
A Job to Create a Test VM
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
ai-bs-vm
13 October 2017 Implementation of CI for Linux Images 13
![Page 14: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/14.jpg)
A Job to Create a Test VM
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
server create
13 October 2017 Implementation of CI for Linux Images 13
![Page 15: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/15.jpg)
A Bare OpenStack VM
Purpose:
� A prerequisite
� A test in its own right
Advantages:
� Faster build
� Faster start
� Faster config
� Extra shortcuts: skip LANDB, DNS, . . .
13 October 2017 Implementation of CI for Linux Images 14
![Page 16: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/16.jpg)
Opportunities for More Shortcuts
Using cloud-init. . .
� . . . to install git
� . . . to git clone tests
� . . .
13 October 2017 Implementation of CI for Linux Images 15
![Page 17: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/17.jpg)
CentOS Functional Tests
https://github.com/CentOS/sig-core-t_functional
� Utilities (bzip2, curl, grep, . . . )
� System bits (kernel, cron, iptables, . . . )
� Services (MySQL, Apache, . . . )
� Compilers (GCC, JDK, . . . )
� File system (root files, . . . )
� X Window
� . . .
13 October 2017 Implementation of CI for Linux Images 16
![Page 18: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/18.jpg)
Sample Runs
13 October 2017 Implementation of CI for Linux Images 17
![Page 19: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/19.jpg)
Jobs To Run Each Test
before_script:
� Install SSH
� Keytab/private key from secret variable
� Wait until we can SSH
� SSH and install git
� Clone CentOS functional tests
What they do:
� SSH and ./runtests.sh $TESTNAME
13 October 2017 Implementation of CI for Linux Images 18
![Page 20: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/20.jpg)
.gitlab-ci.yml
job_template: &job_template
script:
- ssh root@$TESTNODE "./ runtests.sh $CI_JOB_NAME"
p_gcc:
<<: *job_template
stage: development
p_git:
<<: *job_template
stage: development
p_java -1.6.0 - openjdk:
<<: *job_template
stage: development
13 October 2017 Implementation of CI for Linux Images 19
![Page 21: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/21.jpg)
Scaling with Tests
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
server create
13 October 2017 Implementation of CI for Linux Images 20
![Page 22: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/22.jpg)
Scaling with Tests
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
server create
13 October 2017 Implementation of CI for Linux Images 20
![Page 23: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/23.jpg)
Scaling with Tests
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
server create
13 October 2017 Implementation of CI for Linux Images 20
![Page 24: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/24.jpg)
Which Granularity?
� A fresh VM for each CI job
� Problem of destructive tests
� A fresh VM for each test? Really?
13 October 2017 Implementation of CI for Linux Images 21
![Page 25: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/25.jpg)
Test VM Pool
Workflows. . .
� . . . to schedule test VMs in OpenStack project
� . . . to manage VM life cycle
� CI Runner VM instead? But special privileges.
13 October 2017 Implementation of CI for Linux Images 22
![Page 26: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/26.jpg)
Interesting Tests
� Testing setups based on BagPlus
� Access to AFS, EOS, CVMFS, . . .
� Locmap
� RPM health
� lemon-host-check
� collectd
� Attaching Cinder volumes
� Performance
13 October 2017 Implementation of CI for Linux Images 23
![Page 27: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/27.jpg)
A Wider Outreach
13 October 2017 Implementation of CI for Linux Images 24
![Page 28: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/28.jpg)
Allow End-Users to. . .
� . . . run their own tests. . .
� . . . against their own images
13 October 2017 Implementation of CI for Linux Images 25
![Page 29: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/29.jpg)
Allow End-Users to Run Their Tests
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
server create
13 October 2017 Implementation of CI for Linux Images 26
![Page 30: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/30.jpg)
Allow End-Users to Run Their Tests
GitLab CI OpenStack
CC7 - x86_64 [2017-04-06]
SLC6 - x86_64 [2017-04-06]
Koji
ssh ./runtests.sh
koji image-build
UserRepository
MergeRequest
server create
13 October 2017 Implementation of CI for Linux Images 26
![Page 31: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/31.jpg)
User Interface
Merge requests either way
� From a CLI in aiadm?imageci run 'CC7 - x86_64 [2017 -09 -20] ' mysql
� Directly from their git repository?
13 October 2017 Implementation of CI for Linux Images 27
![Page 32: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/32.jpg)
Querying Results
� GitLab API
� Clear views
� From CLI in aiadm?imageci results 'CC7 - x86_64 [2017 -09 -20] ' mysql
Or simply. . .
13 October 2017 Implementation of CI for Linux Images 28
![Page 33: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/33.jpg)
Sample Runs
13 October 2017 Implementation of CI for Linux Images 29
![Page 34: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/34.jpg)
Concerns and Opportunities
� Jobs running for too long
� How many jobs can we run?
� Work to adapt CentOS functional tests
� Testing Docker images
13 October 2017 Implementation of CI for Linux Images 30
![Page 35: Implementation of Continuous Integration for Linux Images · GitLab CI OpenStack CC7 - x86_64 [2017-04-06] SLC6 - x86_64 [2017-04-06] Koji ssh ./runtests.sh koji image-build server](https://reader033.vdocuments.pub/reader033/viewer/2022042305/5ed09ab2e2e54a54101264d4/html5/thumbnails/35.jpg)
Thanks. . .
. . . for the useful chats and suggestions:
Dan van der Ster, Jan Iven, Jan van Eldik,Nacho Barrientos, Raul Garcia,Thomas Oulevey, Vincent Brillault, the ASDFaudience, the LCS Section.
13 October 2017 Implementation of CI for Linux Images 31