nttドコモ様導入事例 openstack summit 2015 tokyo 講演「after one year of openstack cloud...

After One Year of OpenStack Cloud Operation (NTT DOCOMO)

NTT DOCOMO Inc.Ken IgarashiNTT Software

Asako IshigakiNEC

Akihiro Motoki

Ken Igarashi○ Leading OpenStack Project at NTT DOCOMO○ One of the first members of proposing

OpenStack Bare Metal Provisioning (currently called "Ironic") - bit.ly/1stuN2E

Asako Ishigaki○ Engineer, NTT Software ○ Developing OpenStack log collection and

analytics tools.

Akihiro Motoki○ Senior Research Engineer, NEC○ Core developer of Neutron and Horizon.

About Us

Our Project

organization

Scalable Test using 100 nodes

System Design

Recovery Tests(12)

Racking and Cabling

24/7 support(14)

User Support(+x)

2014-6 2014-8 2014-11 2015-2 2015-5 2015-112015-8

o Team Rules (Culture) Focusing on using OpenStack instead of developing OpenStack

Think how to use it. Don’t think OpenStack can’t do XXXX.

Reducing Opex/Promoting Automation Operation tools

• “Anything that a humane needs to do more than twice must be automated.”

Reduce operators by HA and self healing.

o Tools Ansible, Python, Shell Script

• pep-8• Ansible-lint• Install

Spec Writing

Review

Production

+5200+ deployments

(2015)

2000+ patches(2015)

Deployment

Procedure

Operation

HAAutomation

o OpenStack Configuration(http://bit.ly/1DbJPUO) Double redundancies for hardware Triple redundancies for software

VMVMVMVM

MySQL (Galera)

Arbitrator

DB1 DB2

DB3 DB4 VMVMNova

OpenStack APIs

Zabbix

LBLBNeutron Agents

PXE, DNS, DHCP

RabbitMQ

o OpenStack Configuration(http://bit.ly/1DbJPUO) Double redundancies for hardware Triple redundancies for software

VMVMVMVM

MySQL (Galera)

Arbitrator

DB1 DB2

DB3 DB4 VMVMNova

OpenStack APIs

Zabbix

LBLBNeutron Agents

PXE, DNS, DHCP

RabbitMQ

o Deployment CMDB Registration

o Choose playbooks for Ansible Dynamic Inventory

Ansible

o Deployments Common: network, account, logging, Zabbix agent, drivers/firmware x

OpenStack: Nova, Swift, Neutron, ……. x 62 HA Configuration

compileInitial update setup

kernel driver firmware filesystemdevelopment environment

Install HDD Driver

o Operation x 31 Common: process restart, log correction OpenStack Operation: usage, VM migration/backup, user

add/delete/quota change OpenStack Monitoring: health check tools

perhost instance check• Launch instances on given node(s)• boot succeed, instance log• Metadata retrieval, login prompt, SSH access• Optionally, test volume attach and its read/write access

o 2015/10/27 4:40pm - 5:20pm Heian (New Takanawa)

What are operators doing behind the Cloud?

Monitoring System

monitoring

o Monitoring System

Weekday daytime

24h / 365d

VMVM…

VMVMSwiftVMVMCinder

VMVMNova

RabbitMQ

Neutron Agents

Data Bases

Fluentd

Elasticsearch

Zabbix

Kibana

VMVM…

VMVMSwiftVMVMCinder

VMVMNova

RabbitMQ

Neutron Agents Data Bases

Memory CPU Network HDD

General

OpenStack

Monitoring Items Self Healing

1,970 25

3,957 59

o RabbitMQ Configuration

3 node cluster cluster_partition_handling, autoheal

Monitoring Split Brain check:

• “rabbitmqctl eval '[N||{partitions,N}<-rabbit_mnesia:status()].’”

Port Check (5672, 25672) Process Check

• Beam.smp• Rabbitmq-server

At least one node running(1/3)• {Openstack-RabbitMQ:grpsum["HostG-

RabbitMQ","net.tcp.service[tcp,,25672]",last,0].count(#3,0,"eq")}=3

• {OpenStack-RabbitMQ:grpsum["HostG-RabbitMQ","proc.num[beam.smp]",last,0].count(#3,0,"eq")}=3

o MySQL Configuration

4 Nodes + 1 Arbitrator

Monitoring Cluster Check

• wsrep_local_recv_queue• wsrep_local_send_queue• wsrep_flow_control_paused• wsrep_local_commits

Arbitrator

o MySQL Cluster

Master

Galera

recv_queuesend_queue

Commit

Replication

Client

Wait until receive OK from replication

o MySQL Cluster Freeze

Master

Galera

recv_queuesend_queue

Commit

Replication

Client

Wait until receive OK from replication

• Disk Failure: (removed from 😀cluster)

• Disk Speed Throttling : 😢

○ Prohibit some self-healing actions Do not reboot some OpenStack processes

– neutron-plugin-openvswitch-agent Do not reboot network nodes

– loose network reachability (can’t recreate network namespace)

Prohibited Actions while MySQL Cluster Freeze

Solved at Liberty?

All the VMs loose connections

o Throttling happens during DB backup Limit Backup Node

Backup Method

Limit Backup Node

LOCK TABLES FOR BACKUP (online)

1. Take from cluster(Donor/Desynced)

2. DB lock and do backup(FLUSH TABLES WITH READ

LOCK) 3. Return to cluster

(wsrep_desync=OFF)

– wsrep_local_recv_queue– wsrep_local_commits

Log Analytics

Kibana

(1) detect critical system-failure

We have to recover immediately

(2) detect malicious access

We need tonotify users

(3) detect no critical errors

Better to be fixedas soon as possible

(4) find errors/warnings that have no service impact

We want to filter out next time

Purpose of Log Analytics

○ e.g.Logs of a dayTotal:

100 GB, 80M linesSum of critical, error and warning logs:

200K linesThe meaningful logs are more restrictive:

(1) 0 critical failure (2) 0 malicious access(3) 6 non-critical failure (4) 6 ignorable failure

0%0%1%

Breakdown of Logs

CriticalErrorWarningInfoDebugOther

Treasure Hunt in The Ocean of Logs

24%23%49%

OpenStack backend

OpenStack

Operation tools

○ We analyze logs to enhance our black list and white list.○ Logs found in our black list are sent to Zabbix.

Log Analytics Based on White/Black List

---------------

Logs trash

Zabbix Kibana

--------------------

expand

reduce

analyze…

addblack list

white list

Log Server

Network Node

Control Node

Compute Node

How to Adopt Black/White List Using Fluentd

Fluentd

Elasticsearch

zabbix_senderfluentd

• Add “ignorable” flag according to white list

• Put metadata to create graphs from the logs

rsyslog

Zabbix

alerts

Kibana

graph graph

Notify Zabbix according to black list

Log Server

How to Adopt Black/White List Using Fluentd

Fluentd

Elasticsearch

zabbix_senderfluentd

1. syslog10:01 crit: hardware failure

path: syslog rsyslog api.log

timestamp: 10:01 10:03 10:04

severity: crit warn ERROR

item: - ids ignore

source_ip: - x.x.x.x -

message: hardware failure

IDS: from x.x.x.x

invalid request format

3. api.log10:04 ERROR: invalid request format

2. rsyslog10:03 warn: IDS: from x.x.x.x

Zabbix

hardwarefailure

Kibana

IDSgraph

critgraph

Example of Our White List # with Juno

• Count response codes and understand the trend. That’s enough.

^keystonemiddleware\.auth_token \[\-\] Unable to find authentication token in headers$

• This ERROR means user’s operation was denied due to quota.• It has no impact to our system. Should be INFO log?

^nova\.api\.openstack \[[^\]]*\] Caught error: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota\..*$

• This WARNING is caused by presence of SHUTOFF instances.

• It is commonplace condition. Need to be ignored.

^nova.scheduler.host_manager \[[^\]]+\] Host has more disk space than database expected .*$

○ We succeeded in reducing logs to be analyzed. In other words, so many meaningless logs have high log-levels.

Effect of Our White List

Without White List: 160K

With White List: 37

reduce99.98%

We can analyze all logs in 2-3 hours a day!

1 year agoWe couldn’t analyze all logs

in a day

Example of Our Black List

• This message indicates disk problem on • Compute node.

^kernel: \[[^\]]*\] XXXXX.*hardware failure\.$

• Corosync needs cleanup its resources.

^pengine: warning: unpack_rsc_op:Processing failed op monitor for .*$

• Fullbackup of mysql failed once.

^mysql_fullbackup\[\d+\]:\sFailed\sto\sMySQL\sfullbackup.*$

Warning alert

Information alert

Demonstration with Kibana○ 3 dashboards

OpenStack All Logs Error Logs Critical Logs Warning Logs IDS

Trademarks○ Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.○ Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.○ logstash is a trademark of Elasticsearch BV.

o Presentation - Operation 2015/10/27 4:40pm - 5:20pm Heian (New Takanawa)「 What are operators doing behind the Cloud? 」

o Exhibition NEC Booth(H4)

28(Wed.)10:45-13:00,16:30-18:30, 29(Thu.) 　 9:00-14:00 NTT Group Booth(S14)

28(Wed.) 13:15-16:15「 Touch and Feel! NTT DOCOMO’s Cloud Operation 」

contact-cloudpf-ml@nttdocomo.com

NEC NTT

nttドコモ様導入事例 openstack summit 2015 tokyo 講演「after one year of openstack cloud...

Technology

nttドコモ様導入事例 openstack summit 2016 barcelona...

『openstackの導入事例/検証事例のご紹介』...

5g早期実現に向けたnttドコモの活動 -...

voiceless communications technologies - nttドコモ · 26...

法人向けモバイル 2015 0722cut · 2015-08-18 ·...

nttドコモとオイシックス・ラ・大地との協業による新サービス...

ip-based foma voice network - nttドコモホーム foma...

nttドコモホーム...id reader/writer shunji yamanaka...

“企画力・クラフト力・現場力・実現力”...

congress deep dive - openstack days tokyocongress deep dive...

vol16 2 008en - nttドコモ manager hiroshi imamura of...

ドコモの変革とチャレンジ - ntt docomo...©2010...

03 67 特集 - ntt公式ホームページ...

nttドコモ 5g 実験関連論文一覧...nttドコモ 5g...

2008年5月30日...

nttデータ様講演 openstack最新情報セミナー...

nttドコモ様向けlte無線基地局装置img.jp.fujitsu.com/downloads/jp/jmag/vol62-4/paper07.pdffujitsu....

nttドコモの完全子会社化について11月27日にnttドコモ取締役会において、持株による売...

officelink - nttドコモ...title officelink author...

vol24 2 004jp - nttドコモホーム · pdf...

nttドコモ様 導入事例 openstack summit 2015 tokyo 講演「after one year of openstack cloud...

nttドコモ様導入事例 openstack summit 2015 tokyo 講演「after one year of openstack cloud...