乐视云计算 OpenStack Mutil-Region

范德良

乐视 OpenStack 集群

Overview

Overview

Overview Havana(RDO)

Nova Glance

Icehouse(RDO)

Nova Glance Neutron Cinder Ceilometer Keystone Heat Horizon

OS

CentOS-6.5 Kernel-2.6.32

Libvirt-0.10.2 Qemu-kvm-0.12.1.2

OpenvSwitch-2.1.2 Ceph-0.87

Overview

全球

数十

个集

群

Icehouse & Havana

Nova-network & Neutron

Public & Private network

Goal: 一次登录，管理全球所有 Region

Drawbacks 用户租户信息不统一

访问集群麻烦

增加运维难度

Scaling

Regions Cells Availability zones Host aggregates

Use when you need

Discrete regions with separate API endpoints and no coordination between regions.

A single API endpoint for compute, or you require a second level of scheduling

Logical separation within your nova deployment for physical isolation or redundancy.

To schedule a group of hosts with common features.

Example

A cloud with multiple sites, where you schedule VMs to a particular site and you want a shared infrastructure.

A cloud with multiple sites where you can schedule VMs "anywhere" or on a particular site.

A single-site cloud with equipment fed by separate power supplies.

Scheduling to hosts with trusted hardware support.

Overhead

A different API endpoint for every region. Each region has a full nova installation.

Considered experimental. A new service, nova-cells. Each cell has a full nova installation except nova-api.

Configuration changes to nova.conf.

Configuration changes to nova.conf.

Shared service

Keystone Keystone nova-api

Keystone All nova services

Keystone All nova services

Scaling

Region 1 Region N

Keystone

Horizon

Api Cell

Cell 1

Nova-* Nova-cell AMQP/DB

AZ 1 AZ N

Nova-compute

Nova-compute

Nova-compute

Nova-compute

Cell N

Endpoint Design

Type:

https:// + service + region + domain + version

• service: compute image identity network … • region: country + city + id cn-bj-1 us-la-2 • domain: lecloud.com

Egg: https://identity.lecloud.com/v3

https://compute.cn-bj-1.lecloud.com/v2

https://volume.cn-sh-2.lecloud.com/v2

推荐 Keystone V3

• 支持 domain，实现子账户功能 • 兼容性 & 升级

Token Format

UUID PKI

openssl cms sign

UUID vs PKI

UUID PKI(PKIZ)

curl -X GET -H "X-Auth-Token: $TOKEN" -H "X-Auth-Project-Id: service" -H "User-Agent: python-novaclient" -H "Accept: application/json" 'https://compute.cn-bj-1.lecloud.com/v2/{tenant_id}/servers'

UUID vs PKI

那么问题来了，Http 服务器默认限制 Header size

若 Region 数量较少(<=3)，推荐使用 PKI(PKIZ) 若 Region 数量较多(>3)，推荐使用 UUID

Apache2 Nginx Python Http Haproxy

8k 4k 16k 4k

Private Net(北京)

Havana

Icehouse

Havana

Icehouse

Keystone

Net

Dashboard

• Keystone、Dashboard 全局唯一

• Keystone 部署于公网，能被各个 Region 访问

• Dashboard 部署于内网, 能访问各个 Region

Public Net (香港，台湾，北美…)

Region Deployment

Compatibility

Tempest 完成兼容性测试 • Keystone

Icehouse & Juno 具有优良的向下兼容性

• Horizon

Python-django-openstack-auth >= 1.1.7

使用 Memcache 存储会话信息

Rally 性能测试

Tempest 集成测试

Unittest 单元测试

Http server

tuning

Apache VS Python server

Keystone 部署于 apache，充分利用多核高并

发优势

Haproxy & Keepalived

Haproxy 提供负载均衡，支持 Keystone

Server 横向扩展，实现高并发

Keepalived 支持 VIP 漂移，防止 Haproxy 单

点故障，实现 HA

Performance

VIP

Haproxy / Keepalived Httpd / Memcached

Keystone

DB Cluster

Https

Haproxy / Keepalived Httpd / Memcached

Keystone

Apache vs Python Http Server

Security

• Haproxy 提供唯一的公网访问入口，仅开放 443 端口，根据匹配 url 转发至 OpenStack 相关服务 。 • HTTPS Haproxy >=1.50 • 防止 DDOS 等攻击

TCP syn flood attacks

Slowloris like attacks

Limiting the number of connections per user

Limiting the connection rate per user • White list