openstack tempest and rest api testing

OpenStack Tempest and

REST API Testing

Agenda

One

• Introduction

• OpenStack & DevStack

Two

• Tempest & Architecture

• DevStack config and H/w required

Three

• Various tests & Test Results

• Key Findings

Four

• Summary & Conclusion

• Tech-Scenarios & Citations

To demonstrate

starting/running

of OpenStack

services

Specifically for

development

work on

OpenStack

We can

download the

latest version of

DevStack from

”git” repo.

Cloud operating

system/platform

that controls large

pools of compute,

storage, and

networking

resources

Used for

developing private

clouds

• Keystone

• Cinder

• Glance

• Neutron

• Horizon

• Nova

OpenStack & DevStack

OpenStack

DevStack

OpenStack Testing ??

OpenStack integration Test Suite

• Tempest Test Suite

• Ready to deploy

Unit Tests in each project

• Nova, Cinder, Swift

• Neutron etc

• OpenStack Code

base is written in

Python

• So how to

use/write tests?

• Isolate the

working

Environment

• Quick testing

• Automated

configuration

• Ease of use

Tempest

• Tempest is a

set of

integration

tests

• To be run

against any

live

OpenStack/

Dev Stack

cluster

F

e

a

t

u

r

e

s

Official OpenStack

Integration Test Suite

Interacts with majority

ReSTful API’s

Around 3000+ tests

covering all projects

Used as gating tests for all

new projects

Runs on all OpenStack

releases

Self Testing/Self Cleaning

Uses a unified tempest

REST client

Latest version of “Tempest

12.1.1.dev37”

A

t

t

r

i

b

u

t

e

s

Scenario based Tests

CLI Tests Stress tests

Ensure that OpenStack cloud

works with the OpenStack API

Basic Tempest Reference Architecture

OpenStack/ DevStack

deployment

Service

Clients

Python

Clients Boto Tests

Rest Client

API TestsScenario

Tests

Third party

tests

CLI based

Tests

LibraryCLI

OpenStack ReST API’s EC2 API’s

OpenStack ReST API’s

1

Basic Tempest Reference Architecture

Compute

Node

Compute

Node

Controller

Node

Test Server

(Tempest Node)

HTTP Request HTTP Response

OpenStack configuration{Tempest tests executed

from controller node}

2

Tempest in a “Nutshell” 1

Tempest in a “Nutshell” 2

Hardware configuration required and OS used 1

Hardware configuration required and OS used 2

Tempest Execution….steps

Test Results-Kilo

968

724

962

22

76

13

141 126 141

1131

926

1116

0

200

400

600

800

1000

1200

CentOS Ubuntu Fedora

toxefull

Passed Failed Skipped Total TC

Test Results-Liberty

999 1023

944

10 5 24

136 136 138

1145 11641106

0

200

400

600

800

1000

1200


toxefullPassed Failed skipped Total TC

Test Results-Kilo

41

32

43

3

9

1

39 38 39

83

79

83

0

10

20

30

40

50

60

70

80

90


toxesmokePassed Failed Skipped Total TC


44 4543

1 02

39 39 39

83 84 84

0

10

20

30

40

50

60

70

80

90


toxesmoke

Passed Failed skipped Total TC

Test Results-Kilo

867

700

1015

55102

15

102 96 107

1024

898

1137

0

200

400

600

800

1000

1200


run_tempest

Passed Failed Skipped Total TC


894

955984

5126 15

138 139 139

1024

898

1137

0

200

400

600

800

1000

1200


run_tempest

Passed Failed skipped Total TC

Key Findings-Failed Tests analysis

OS/Tests toxefull toxesmoke run_tempest

CentOS 7.0 Most of the Tempest test case

failures are

• Volume API’s

• Third party boto (this is related

to Elastic Compute (EC2) and

Object storage-S3 features of

AWS) ReST API failures.

Most of the Tempest test case

failures are

• Volume API’s related

Most of the Tempest test case failures are

• Scenario tests, snapshot pattern related,

Volume API’s and third party boto(this is

related to Elastic Compute (EC2) and

Object storage-S3 features of AWS)

ReST API failures.

Ubuntu 14.04 Most of the Tempest test case

failures are

• Compute, scenario and Cinder

Volume related

• Third party boto (this is related to

Elastic Compute (EC2) and



Most of the Tempest test case

failures are

• Compute ,scenario and

Volume Rest API failures.


• Compute, scenario, Volume and third

party boto (this is related to Elastic

Compute (EC2) and Object storage-S3

features of AWS) ReST API failures.

Fedora 21 Most of the Tempest test case

failures are

• Volume API’s

• Third party boto (this is related

to Elastic Compute (EC2) and



• Only one of the test case

failed here and is related to

Volume ReST API failure.


• Volume API’s and third party boto (this is


Object storage-S3 features of AWS)

ReST API failures.

Kilo

Key Findings-Failed Tests analysis

OS/Tests toxefull toxesmoke run_tempest

CentOS 7.0 Most of the Tempest test case

failures are

• Volume API’s

• Third party boto (this is related to



AWS)

• Server multi-node ReST API

failures.

Tempest test case failure is

related to following

• Volume API’s


• Compute, Volume API’s, basic-Ops test

server, multi-node

• third party boto(this is related to Elastic

Compute (EC2) and Object storage-S3

features of AWS) ReST API failures.

Ubuntu 14.04Most of the Tempest test case

failures are

• third party boto” (this is related to




No failures found. Most of the Tempest test case failures are

• Compute, scenario, Volume and third party

boto (this is related to Elastic Compute

(EC2) and Object storage-S3 features of


Fedora 21 Most of the Tempest test case

failures are

• scenario tests, compute,

snapshot deleting, Volume API’s

and third party boto (this is

related to Elastic Compute (EC2)

and Object storage-S3 features

of AWS) ReST API failures.

Found two failures which are

• Volume ReST API failure.


• Volume API’s and third party boto (this is


Object storage-S3 features of AWS) ReST

API failures.

Liberty

Key Findings-Skipped Tests analysis

Some of the major reasons for “Skipped” tests are,

Pending bugs need to be resolved

Live block migration not supported

Instance validation and multiple instance support not available

SSH & VNC support not available in some cases

Some of the features not included in Dev Stack

Ephemeral disk verification could not be done

Multiple policy feature not supported

Instance validation and multiple instance support not available

Note:- These findings are across OpenStack releases and OS flavors

Liberty

Kilo

Key Findings-SummaryKILO LIBERTY

toxefull toxefull

Passed Failed Skipped Total TC Passed Failed skipped Total TC

Centos 968 22 141 1131 CentOS 999 10 136 1145

Ubuntu 724 76 126 926 Ubuntu 1023 5 136 1164

Fedora 962 13 141 1116 Fedora 944 24 138 1106

toxesmoke toxesmoke


Centos 41 3 39 83 CentOS 44 1 39 83

Ubuntu 32 9 38 79 Ubuntu 45 0 39 84

Fedora 43 1 39 83 Fedora 43 2 39 84

run_tempest run_tempest


Centos 867 55 102 1024 CentOS 894 51 138 1024

Ubuntu 700 102 96 898 Ubuntu 955 26 139 898

Fedora 1015 15 107 1137 Fedora 984 15 139 1137

Conclusion

• Tempest test results across different OpenStack releases and OS flavors is distinct

• One of the primary reasons of this ambiguity is each OS flavor having different

kernel versions.

• The more latest the Kernel version used, the more stable are the test results

• The other reason is because of different OpenStack releases on different hardware

platforms

• Tempest execution on Liberty release on various Linux OS flavors is yielding more

stable test results as compared to Kilo release.

• The more latest the OpenStack version, the more stable are the test results.

• These test results can be used for future reference (based on h/w architecture used)

Troubleshooting…..

1. We have observed some times that when “localrc” file from root user and try to run

stack.sh command from the stack user, following error will be encountered.

Solution:- When such error is encountered, change the ownership of devstack

directory to stack user using the following command

Troubleshooting…..

2. If you are using proxy, While running stack.sh command you may get the

following error

Solution:- Best practice is to have internet connection without proxy, which will

avoid majority of the issues.

3. When we clone “nova” manually to /opt/stack/nova from git repository and try to

execute the command “stack.sh”, following exception will be encountered.

Solution:- To avoid the above exception, copy .git directory from cloned devstack

directory to /opt/stack/nova directory. This will help us to overcome the above

mentioned exception.

Tech-Scenarios

Tech-Scenario-I Validation of OpenStack workload management tool

Context

Validation and functionality test of

OpenStack & management tool together

was required

Validation of RESTful API’s for supported

OpenStack releases was required

Certification testing on different OS

platforms supporting OpenStack was

required.

Solution/approach used

Used Tempest test suite (along with some

customization where required) for the

validating stability of ReSTful API’s

Using Tempest test suite and some home

grown test cases, we were able to verify the

management tool and various OpenStack

features

Triaging with multiple combinatorial tests

and identifying gaps and helping respective

customer to fix the issues on time.

Outcomes & Learning

Since OpenStack was niche technology,

understanding the requirement clearly was a

challenge

State of the art certification of RHEL and

Ubuntu on OpenStack was a challenge.

Support for Continuous OpenStack testing

for supported releases was a challenge

Technical Challenges

Technology Ubuntu, SuSE, RHEL, Cent OS , OpenStack releases Havana, Icehouse and Juno

Test Suite used Tempest

Validation of complete workload management

tool on OpenStack with various test scenarios

Validating Nova, Keystone, AMQP message

bus protocol and Horizon interfaces.

Evaluation of the management tool for

robustness and stability of usage.

Triaging issues that are related to OpenStack

product

Certification of various OpenStack releases

done for RHEL and Ubuntu.

Ready test suite for future releases of

OpenStack was evolved

Tech-Scenario-II: Validation of OpenStack API’s using Tempest suite

Context

Complete coverage of OpenStack API

testing required

Reliability and Serviceability of OpenStack

on a specified hardware configuration was

required

Solution/approach used

Validation of complete ReST API suite for

various OpenStack releases and for specified

h/w platforms.

Complete test failure analyses with log

reporting on time.

Parallel Tempest configurations kick started

with test executions

Outcomes & Learning

Running Tempest test suite on multiple

Linux platforms

Debugging and finding failures and causes

Rally configuration and execution

Technical Challenges

Technology Ubuntu, RHEL, Cent OS , OpenStack releases Juno and Kilo,

Test Suite used Tempest

Tools Rally

Tempest configuration on multiple

platforms and OpenStack releases.

Validated complete basic OpenStack API

suite and reported failures

Suggested fixes where possible

Citations

http://docs.openstack.org/

https://github.com/openstack/

http://docs.openstack.org/developer/tempest/field_guide/

https://wiki.openstack.org

https://wiki.openstack.org/wiki/devstack

http://docs.openstack.org/developer/devstack/overview.html

http://devstack.org

http://devstack.org/faq.html

https://github.com/openstack-dev/devstack

http://docs.openstack.org/developer/tempest/configuration.html

http://docs.openstack.org/

https://github.com/openstack/

http://docs.openstack.org/developer/tempest/field_guide/

https://wiki.openstack.org/

https://wiki.openstack.org/wiki/devstack

http://docs.openstack.org/developer/devstack/overview.html

http://devstack.org/

http://devstack.org/faq.html

https://github.com/openstack-dev/devstack

http://docs.openstack.org/developer/tempest/configuration.html

Thank You

Shashidhar SoppinSenior Architect

Reachable @

“[email protected]”

Back-up Slides

What is Tox?

tox is a generic virtual environment management and test command line tool

that we can use for the following purposes.

We can check our package installed correctly with different Python

versions and interpreters or not.

Running our tests in each of the environments, configuring our test tool

of choice is possible with using tox.

tox acts as a frontend to Continuous Integration servers.

tox will come default along with DevStack package. But if you want to

manually install and configure it, install tox with pip install tox. Then put

basic information about the test environments you want to carry out

into a tox.ini file and follow the tox execution sequences as explained

earlier.

How to download latest Devstack?