design and implementation of automatic system …failure scenario 5 selection of recipes from db...

Design and Implementation of Automatic System for Network Testing with Quality Degradation

Sep. 27, 2017, Seoul, Korea

Junichi Kawasaki,

Megumi Shibuya, Atsuo Tachibana, Masanori Miyazawa and Teruyuki Hasegawa

KDDI Research, Inc.

1. Introduction

Network Testing

SDN/NFV

×

Service Design Configuration Testing S-inLegacy

SDN/NFV Test items

Manual-based approach- cannot cover all items- human error risk

-To maintain SLA between customers and carriers

-When devices are added, removed, upgraded

-Over hundreds of test items

-Virtualized: More components to be tested

-Diversified services: More patterns to be tested

-Agile implementation: Shorter time for testing

2. Our target and proposal

Policy-based path control-Becoming widely used

(e.g., hybrid WAN)

-Monitor and switch paths

in responding to network

conditions

Testing automation

CloudShell/TestShell-Protocol testing

-Protocol: BGP/OSPF

Does not support testing

in degradation cases

Automatic System for Network Testing with

QualityDegradation

Testing with various quality patterns is needed

3. Automatic system for network testing

NtbT Server

Chef server Zabbix server

, : Virtual Bridge

KVM

OSS DB (testing scenarios,

NtbT topologies, recipes,

logs)

NtbT

(NW to be tested)

Web applicationTesting Control Server

vSW vSWvSW vhost vhost

Physical-SW

-Create test scenarios

-Check results

: Testing Control Plane

: Data Plane

Key Functions for automatic testing

➢ Automatic construction of NtbT

➢ Automatic failure occurrence

➢ Monitoring network status

3.1 Testing scenario

Procedure:

I. Create a testing scenario with start time

II. Scenario runs automatically

III.Check the monitoring result

Testing Scenario No. Scenario Step

NtbT

construction

scenario

1 Selection of NtbT topology from DB

2 Construction of NtbT

3 Initialization of network equipment

4 Checking of connectivity

Failure scenario

5

Selection of recipes from DB with execution time

Traffic recipes : traffic sending/receiving

Failure recipes: interface down/up

latency start/stop

:

IP version

Bandwidth, etc.Node

Interface, etc.

Monitor network

status during testing

3.2 Automatic construction of NtbT

DB

OSS

(Chef, Zabbix)

Testing Control Server

: Chef client

: Chef server

NtbT Server

Construction

Scenario

: Data NW

: Control NW Virtual switch

with Chef client

installed

Create VM

=>Configure

=>Check connectivity

DBJSON-

based

Config.Hostname

Control IF

Data IF,

etc.

3.3 Automatic failure occurrence

DB

OSS

(Chef, Zabbix)


NtbT Server

Failure

Scenario

delay

: Chef client

: Chef server

: Data NW

: Control NW

Failure

Recipe Node

IF

Degradation type

Degradation values

Failure

Recipe1SW2

FE0/1

Delay

3000ms

Failure

Recipe2SW2

FE0/1

Delay

4000ms

SW2

FE0/1

Delay

1000ms

Fluctuating

Quality

Failure

Recipe3

3.4 Monitoring network status

snmpd

Testing Control Server vSW, vhost

Traffic volume

Log

DB

iperf

vhost

End-to-End quality(speed, jitter, loss)

zabbix_

sender

zabbix_

server

zabbix_

agentd

Every 10 seconds

Ipef sends/receives/

analyzes packets

every second

↓

Zabbix_sender

uploads periodically

4.1 Network topology for system evaluation

Monitor and Switching

Measured delay(s) State

delay<x NORMAL

x≦delay<y DELAY

y≦delay DOWN

※Threshold x, y is configured on GW.

Monitor & switching are

enabled on GW1-CE1

and GW3-CE1

The path of a state better than another becomes the working path

4.2 Network implementation

OSS

(Chef, Zabbix)


NtbT Server

DB

ProxyProxy

Proxy

Proxy

Proxy

Proxy

PC1CE1

L3SW

GW1

GW3

RR GW2 CE2 PC2

OSS client

cannot be

installed

Proxy

for CE2

snmpd/

zabbix_agentd

: Chef client

: Chef server

: Data NW

: Control NWTotal 7 proxy nodes

on the testbed

OSS client

installed for

automation

Testing Control

ServerNtbT Server

OS Ubuntu 14.04

Memory 64GB

CPU 28core 28core

Chef Server 12.4.0 Client 12.5.1

Zabbix 2.4.8 (server) 2.2.2 (agent)

System environment

4.3 Failure scenario for system evaluation

Recipe

No.

Failure Recipes

[x = 3, y = 5]

1 Cause 2 sec delay on the link between GW1-CE1

2 Change the delay with 4 sec


4 Normalize the delay

5 Cause 2 sec delay on the link between GW3-CE1



8 Normalize the delay

GW1-CE1 GW3-CE1Working

path

NORMAL NORMAL A

DELAY NORMAL B

DOWN NORMAL B

NORMAL NORMAL B

NORMAL NORMAL B

NORMAL DELAY A

NORMAL DOWN A

NORMAL NORMAL A

Verify with GW log

Verify with traceroute (manual) or

visualized traffic on system GUI (with system)

Expected Results

4.4 Evaluation results

Perform the scenario manually and, with the proposed

system, twice, respectivelyStep Manual operation With system

1 Execute command Create scenario

2 Check result Run scenario

3 Repeat 1-2 Check result

Case Testing OperationAverage

Time[min]

Average

Work Hours[min]

Case-1 Manual operation 21:05 21:05

Case-2 With system 27:51 11:12

Results

Reduced

46.9%

Operator’s

work hours

5. Conclusion

Proposal • Automatic system for network testing with quality

degradation

Implementation of automatic system• Construction of NtbT and failure occurrence by Chef

• Monitoring network status by Zabbix

System evaluation• Testing on experimental network that consists of IP-Sec GW

• Reduce testing workload by half

• No human error risk

Ref.1) Creating a testing scenario

Execution

timeRecipe Node Parameter

Traffic receiving start

Traffic sending start

Degradation start

Degradation change

Degradation change

Interface

Traffic parameter

(IP version,

bandwidth etc.)

Degradation parameter

(Loss rate,

Latency values etc.)

Ref.2) The scenario running

design and implementation of automatic system …failure scenario 5 selection of recipes from db...

Documents