vsphere平台上运行企业级...

vSphere平台上运行企业级J 应用的最佳实践Java应用的最佳实践

连冠华

2011年10月26日

1

© 2010 VMware Inc. All rights reserved

议程

传统的企业级Java平台传统的企业级Java平台

VMware vSphere™企业级Java平台体系架构

设计和扩展设计和扩展

最佳实践

高可用性高可用性

从物理机迁移到vSphere

故障诊断和性能调优故障诊断和性能调优

成功案例分享

Q&A Q&A

2

传统的企业级Java平台

企业级应用是多层的企业级Java应用是多层的

当有任何变化发生的时候，多层的复杂性将会导致连锁反应

满足多部门应用的需求导致了成本的增加满足多部门应用的需求导致了成本的增加

DB ServersJava Applications

Load Balancer Tier

Load Balancers Web Servers

Web Server Tier Java App Tier DB Server Tier

pp

IT Operations IT Operations IT Apps – Java IT Ops & Apps pNetwork Team

pServer Team

ppDev Team

p ppDev Team

Organizational Key Stakeholder Departments

3

vSphere上企业级Java平台的体系架构

DB ServersJava ApplicationsLoad balancers Web Servers

High Uptime, Scalable, and Dynamic Enterprise Java ApplicationsLoad Balancers as VMs

Web Servers

APPLICATION SERVICES

VMware vSphere

Capacity On Demand High AvailabilityDynamicJava Application Servers

SHARED INFRASTRUCTURE SERVICES

SHARED, ALWAYS‐ONINFRASTRUCTURE

4

云时代的应用程序基础架构

VMware vFabric 云应用程序平台

框架和工具 Rich Web Integration Batch Data Access Social /Cloud Tool Suite

Dynamic Load B l

Performance M t

Policy-driven A t ti

Elastic A S

Cloud M i

Global Data M tt S ERS (A h ) H iR bbitMQG Fi

vFabricBalancer Management AutomationApp Server MessagingManagementtc Server ERS (Apache) HypericRabbitMQGemFire

平台服务

虚拟数据中心云基础设施和管理

5

在vSphere上设计和规划企业级Java应用

Step 1 –

Establish Load profile

Step 2

Establish Benchmark

Step 3 –

Size Production Env.

From production logs/monitoring reports measure:C t

Iterate through Benchmark test until you are satisfied with the Load profile

The size of the production environment would

Concurrent UsersRequests Per Second

pmetrics and your intended SLAafter each benchmark iteration

have been established in Step2, hence either o roll o t theSecond

Peak Response TimeAverage

benchmark iteration you may have to adjust the Application Configuration Adj t th S h

you roll out the environment from Step‐2 or build a new one based on

Response TimeEstablish your response time SLA

Adjust the vSphereenvironment to scale out/up in order to achieve your desired

new one based on the numbers established

number of VMs, number of vCPU and RAM configurations

6

在vSphere上设计和规划企业级Java应用

第一步：建立负载计划

第二步：建立性能基准

第三步：规划生产环境

从生产环境的日志，监控报告等评估

进行迭代的性能基准测试直到你对你的负载计划和期望

生产环境的基准和规划在第二步就应该已经建立因监控报告等评估：

并发用户数每秒并发请求高峰时期响应

的负载计划和期望的SLA满意为止

每次基准测试你都

应该已经建立，因此你可以直接从第二步建立的基准虚

高峰时期响应时间

平均响应时间

可能需要调整应用的配置参数

拟机批量部署或者从已经部署的虚拟机上建立一个新的

建立你的响应时间SLA

调整vSphere环境参数横向或纵向扩展以达到你所期望的虚拟机 CPU

机建个新的基准。

的虚拟机，vCPU，内存的配置

7

第二步– 建立性能基准

Investigate bottlneckedlayer

Network, Storage, Application Configuration,

ESTABLISH BUILDING BLOCKESTABLISH BUILDING BLOCK VM

Establish Vertical scalabilityScale Up Test

est

Application Configuration, & vSphere

If l tIf scale outbe in terms of vCPU and memory

Establish how many JVMs on a VM?Establish how large a VM would be in terms of vCPU and memory

cale

Up

Te

If scale out bottlenecked

layer is removed,

iterate scale out t t

If building block app/VM configproblem, adjust

& iterate No

Building Block VMS

c Building Block VM

SLA

Test complete

testtest

DETERMINE HOW MANY VMDETERMINE HOW MANY VMsEstablish Horizontal Scalability

Scale Out TestHow many VMs do you need to

Scale Out Test

SLA OK?

How many VMs do you need to meet your Response Time SLAs without reaching 70%-80% saturation of CPU?Establish your Horizontal

Building Block VM

Building Block VM

8 application

scalability Factor before bottleneck appear in your application

vSphere上规划和设计HotSpot and jRockit JVMs

VMGuest OSMemory

VM Memory

‐Xss per thread

JVM Memory

Java Stack

Other memy

Perm Gen ‐XX:MaxPermSize

JVM JVM MaxHeapHeap‐Xmx Initial

Heap‐Xms

9

vSphere上规划和设计HotSpot and jRockit JVMs

G t OS M 1G (d d OS/ th ) Guest OS Memory approx 1G (depends on OS/other processes)

P Si i ddi i l h X (M H ) l d

VM Memory = Guest OS Memory + JVM Memory

JVM Memory = JVM Max Heap (‐Xmx value) + JVM Perm Size (‐XX:MaxPermSize) +

Perm Size is an area additional to the –Xmx (Max Heap) value and is not GC-ed because it contains class-level information.

NumberOfConcurrentThreads * (‐Xss) + “other Mem”

“other mem” is additional mem required for NIO buffers, JIT code cache, classloaders, Socket Buffers (receive/send), JNI, GC internal info

If you have multiple JVMs (N JVMs) on a VM then:• VM Memory = Guest OS memory + N * JVM Memory

10

vSphere上规划合设计IBM JVM

VMGuest OSMemory

VM Memory

‐Xss per thread

JVM Memory

Java Stack

Other memy

Perm Gen ‐XX:MaxPermSize

Perm Gen

JVM JVM MaxHeapHeap‐Xmx Initial

Heap‐Xms

11

规划示例

set mem Reservation to

VM

set mem Reservation to 5070m or set to Active

memory used by VM which could be lower

Guest OSMemory

VM Memory(5070m) 500m used by OS

could be lower.

JVM Memory

Java Stack ‐Xss per thread (192k*4)

Other mem (=217m)(4570m) Perm Gen ‐XX:MaxPermSize (256m)

JVM MaxJVM MaxHeapXmx

Initial Heap

( )‐Xmx

(4096m)

Heap‐Xms (4096m)

12

vSphere上企业级Java程序最佳实践

使用指导手册使用指导手册Enterprise Java Applications on VMware – Best Practices Guide (http://www.vmware.com/resources/techresources/1087 )

设置内存预留

Set UseLargePages (set at JVM and OS level)

• In jRockit: -XXlargePages

• In HotSpot: -XX:+UseLargePages

• In IBM: -Xlp

Set -Xmx equal to –Xms

Set –Xmn (HotSpot) or –Xns (jRockit) to approx 33% of max heap.

13


打开超线程打开超线程

ESX Host Clusters - When creating Cluster enable VMware HA and VMware DRS

VMware DRS – enables vCenter Server to manage hosts as an aggregate pool of resources.

Use latest Nehalem or Shanghai based hardware

VMs with a number of vCPUs equal to or less than the number of cores in each NUMA node will be managed by the NUMA scheduler and will have g ythe best performance.

The VM-to-Host affinity: honor ISV Licensing, and anti-affinity ensure no 2 HA pairs on the same host.

14


U L d B l th t i t t ll ith S h API Use Load Balancers that integrate well with vSphere APIs

d B l Al i h Ch i A d VM S Load Balancer Algorithm Choice And VM Symmetry

Establish appropriate Thread ratios that Prevents bottlenecks • (HTTP Threads:Java Threads:DB connections)

Googlebot or other web crawling robots somehow traverse to the app, can cause performance degradationpp, p g• Best to divert the robot

15

High Availability Considerations of Enterprise Java Applications

Phased approach to increasing uptime

Ph 3

Establish a Disaster

Phase 3

Recovery Strategy

Phase 2Establish Dynamic

Runtime Mechanism for HandlingTraffic

Establish Zero

Handling Traffic Bursts

Phase 1

Establish Zero‐Downtime

Application Release Strategy

16

企业级Java应用的高可用性考量

增加Uptime的阶段策略

第三阶段

第二阶段建立灾难恢复策略

第一阶段

建立动态运行环境机制处理请求激增

建立零宕机的应用程序发布策略

17

高可用性阶段 1 – 零宕机发布

第步 (发布之前)第一步 (发布之前) 建立一个生产环境 1的拷贝我们称为生产环境 2 你可以使用虚拟机克隆，模板等技术实现Load Balancer

第二步使用负载均衡机制从生产环境1切换到生产环境2

Production 1 Production 2

Step 2Step 5

第三步对生产环境 1的所有VMs建立快照开始使用主部署脚本进行应用的部署和发布过程

Production 1 Production 2

Step 1

第四步同步生产环境1和生产环境2的数据库，因为在生产环

境2提供服务的时候有可能数据已经发生变化

Step 3境2提供服务的时候有可能数据已经发生变化

第五步第五步如果新部署的QA测试过程是成功的，然后从负载均衡

器级别切回到生产环境 1 如果不成功, 可以从第三步建立的快照进行回滚，然后

切换回去

Step 4

18

切换回去


A h A 主发布脚本 Apache Ant主发布脚本

<project name="MyProject" default="runMasterDeploymentScript" basedir=".">

<target name =”runMasterDeploymentScript”Depends=”takeSnapshotOfVM,

switchLoadBalancer, deployDBChangesdeployDBChanges, deployWARFile ”>

</target>

19


主发布脚本A h A 主发布脚本Apache Ant

<target name="takeSnapshotOfVM"><java fork="true" maxmemory="256m"jvm="${JavaHome}"classname="yourcompany.SnapshotManager" ><arg value="-takeSnapshot"/> <arg value=”${vcenterURL}/><arg value= -takeSnapshot /> <arg value= ${vcenterURL}/>

<arg value=”${adminUsername}”/> <arg value=”${adminPwd}/> <arg value=”${vmname}”/> <arg value=”${snapshotname}”/> <arg value=”${desc}”/>

<classpath><pathelement path="${VIJavaHome}/vijava2120100715.jar"/>

</classpath></java>

</target></target>

20


主发布脚本和VI J API 主发布脚本和VI Java APIpublic String takeSnapShotOfVM( String vcenterURL, String adminUsername, String adminPwd, String vmname, String snapshotname, String desc){

ServiceInstance si = getServiceInstance(vcenterURL, adminUsername, adminPwd);Folder rootFolder = si.getRootFolder();String returnFlag = null;try {

VirtualMachine vm = (VirtualMachine) new InventoryNavigator(rootFolder).searchManagedEntity(VIRTUAL_MACHINE, vmname);

Task task = vm.createSnapshot_Task(snapshotname, desc, false, false); if(task.waitForTask()==Task.SUCCESS){

returnFlag = Task.SUCCESS;}else{

returnFlag = "failure";}

} catch (InterruptedException e) {// other catch clauses omitted for brevity…// other catch clauses go here…

}return returnFlag;

21

}


VI J API VI Java API• Getting started tutorial here:

• http://vijava sourceforge net/doc/getstarted/tutorial htmhttp://vijava.sourceforge.net/doc/getstarted/tutorial.htm

• Samples on creating:

• http://vijava.svn.sourceforge.net/viewvc/vijava/trunk/src/com/vmware/vim25/mo/samples/

VM CloneVM,CreateVM,GetUpdates,MigrateVM,RemoveVmDisk,VMClone,VMReconfig,VMSnapshot,VmNicOp,VmRename

Cluster C t Affi it l DRS M ti hi tCluster Create, Affinity rules , DRS vMotion historyStorage AddDatastore,CopyFile,PrintStorageSystem,QueryVirtualDiskNetwork AddDVS,AddNIC,AddVirtualNic,AddVirtualSwitch,AddVirtualSwitchPortGro

up FlipNetworkService RemoveVirtualNicup, FlipNetworkService, RemoveVirtualNic

22

高可用性阶段 2 – 建立动态的高可扩展性

虚拟机的垂直可扩展性 CPU和M 的热添加虚拟机的垂直可扩展性 – CPU和Memory的热添加

• 提前计划和打开这个特性

• 虚拟机必须关闭才能打开热添加功能在打开这个功能后只要操作系统支持热• 虚拟机必须关闭才能打开热添加功能。在打开这个功能后，只要操作系统支持, 热添加CPU和热添加内存可以在运行时完成，而不需要关闭虚拟机。

Scalab

ility

Vertical S

Add CPU/Memory/Storage

23


如果想打开热添加功能如果想打开热添加功能:Go to VM > Edit Setting > Options tab

24


25


Horizontal Scalability

Add more Hosts

Horizontal Scalability

26


DRS DRS

27


负载均衡器集成负载均衡器集成

• 一些负载均衡器提供API和VMware vCenter™ Server进行集 Load Balancer

成. 虚拟机添加的时候vCenter会知道这个变化，因此它可以在运行时调整资源池的配置。 Newly added VMs

• F5 for example has the iControl API

Newly added VMs automatically added to load balancer

• http://www.f5.com/solutions/applications/vmware/

vsphere/

• Graceful Server Shutdown• Graceful Server Shutdown, • http://devcentral.f5.com/Tutorials/TechTips/tabi

d/63/articleType/ArticleView/articleId/254/iContro

l-Apps--04--Graceful-Server-Shutdown.aspx

28

高可用性阶段 3 – 灾难恢复

Global Load Balancer Global Load Balancer

Load Balancer Load Balancer Load Balancer Load BalancerLoad Balancer Load Balancer Load Balancer Load Balancer

SRM Array Replication App clusterApp cluster Replication

Datacenter Location‐1 Datacenter Location‐2

DRS Cluster DRS Cluster

29

移植到vSphere

分阶段虚拟化分阶段虚拟化

• 阶段 1 – 虚拟化Web Server层

• 阶段 2 虚拟化Java 应用服务器层• 阶段 2 – 虚拟化Java 应用服务器层

• 阶段 3 – 虚拟化数据库服务器层

• 阶段 4 – 虚拟化负载均衡器层

Enterprise Java Applications with all Tiers Virtualized

Phase 2Phase 1 Phase 3Phase 4

DB ServersJava ApplicationsLoad balancers Web Servers

30

移植到vSphere

关键考量关键考量

• 在UNIX上面, UseLargePages标志默认情况下是打开的,因此你也必须在Linux上打开改选项因为使用大页面可以显著提升性能。

• 当你把你的Java应用移植到Linux上时，你可以使用如下设置:• Set the –XX:+UseLargePages at the JVM level for Sun HotSpot• Set the –XXlargePages at the JVM level for JRockit • Set the huge pages at OS level, refer to:

• http://www.vmware.com/resources/techresources/1087p• http://www.vmware.com/files/pdf/large_pg_performance.pdf

植当你从32位的JVM移植到64位的JVM:• 好处是你可以使用更大的Max Heap设置

• 然而相同32 bit的应用程序需要更多的内存去运行在64 bit的平台上因为需要更• 然而, 相同32-bit的应用程序需要更多的内存去运行在64-bit的平台上因为需要更多的指针空间

• 在Hotspot上你可以使用UseCompressedOops最大到32GB空间

31

性能层面

• See the Performance of Enterprise Java Applications on VMware vSphere 4.1 and SpringSource tc Server at http://www.vmware.com/resources/techresources/10158 . p

32

性能层面

在2 CPU物理机和虚拟机的对比测试案例中百分之90的响应时间曲线和在2 CPU物理机和虚拟机的对比测试案例中，百分之90的响应时间曲线和CPU的利用率

在虚拟机的CPU利用率低于80%的情况下本地和虚拟化的配置拥有基本在虚拟机的CPU利用率低于80%的情况下, 本地和虚拟化的配置拥有基本一致的性能表现，只在响应时间的绝对值上有微小的差异.

% CPU% CPU

R/T

80% Threshold

33

性能层面

Lowest CPU %

Best R/T

Number of vCPUs per VM

Number of VMs

Per-VM Maximum Heap Size

Total Heap for 4vCPU Case

per VM Size1 4 2GB 8GB2 2 2.5GB 5GB Best case

34

4 1 4GB 4GB

Cardinal Health Java VMon VMware

35


关于Cardinal Health

创建于1971

业界领先的医疗产品和服务提供商业务范围涉及医疗供应链的所有业界领先的医疗产品和服务提供商，业务范围涉及医疗供应链的所有环节;通过多种渠道开展业务

每天横跨北美在60,000个医疗点为超过50,000名客户提供服务

在美国和波多黎各有大约超过三分之一的药剂，医疗实验室产品和医疗产品都通过Cardinal Health供应链来提供

全球超过30 000 名员工在超过10个国家直接运营业务全球超过30,000 名员工;在超过10个国家直接运营业务

《财富》杂志美国公司500强排名第19位

The business behind healthcare with the broadest view of the supply chain

36

虚拟化

虚拟化之旅

时间 2005 – 2008 2009 – 2011 2012 – 2015时间 2005 – 2008 2009 – 2011 2012 – 2015

主题IT集中化管理共享服务

固定成本密集

快速响应

变化成本

服务定制

Consolidation < 40% Virtual <2,000 VMs

Internal cloud >58% Virtual >3,852 VMs

Cloud Resources • >90% Virtual >8,000 VMs

Virtual

时间 2005 2008 2009 2011 2012 2015时间 2005 2008 2009 2011 2012 2015

, <2,355 physical

Data Center Optimization 30 DCs to 2 DCs

, <3,049 physical

Power Remediation P2Vs on refresh

, <800 physical

Optimizing DCs Internal disaster recoveryDC 30 DCs to 2 DCs

Transition to Blades

P2Vs on refresh

HW Commoditization

Internal disaster recovery Metered service offerings

(SAAS, PAAS, IAAS)Shrinking HW Footprint

50% Utili ti

DC

<10% Utilization <10:1 VM/Physical

Low Criticality Systems

15% Utilization 30:1 VM/Physical

Business Critical Systems

> 50% Utilization > 60:1 VM/Physical

Heavy Lifting Systems

HW

y y 8X5 Applications

y SAP ~ 382 WebSphere ~ 290 Unix to Linux ~ 655

Database ServersSW

3737

虚拟化为什么在Vmware平台上虚拟化WebSphere？

数据中心策略一致性

• 池化资源容量 ~15% 利用率

• 弹性 – 用户改变工作负载

• Unix到Linux迁移，去小机化

• 灾难恢复• 灾难恢复

简化和管理

• 实现成千上万应用的高可用性而不是成千上万的高可用解决方案实现成千上万应用的高可用性而不是成千上万的高可用解决方案

• DMZ区的网络和系统管理

五年节省600万美元五年节省600万美元

• 硬件节约 ~ $660K

• WAS Licensing ~ $862K

• Unix to Linux ~ $3.7M

• DMZ – ports~ >$1M

3838

性能 & 可扩展性

IBM DB2 性能

I/O is not a bottleneck for application servers or even reasonable DB servers

39

高可用性降低复杂性

实现成千上万应用的高可用性而不是成千上万的高可用解决方案

Challenges: • Affinity/Anti-affinity • Complexity • User Error• No SPOFs

4040

IT的转换之旅 – 加速和增强

云低管控高管控

企业级混合云

基础架构位中心应用位中心业务为中心

虚拟化

业务生产 IT 即服务IT 生产

41

谢谢！


vsphere平台上运行企业级...

Documents