Reducing the Operations Burden andIncreasing QoS with

WSO2 Platform Managementand Administration Tactics

Sanjaya RatnaweeraAmani Soysa

Agenda

● Common Deployment Patterns● Deployment Synchronization● Registry Mounting● Setting up a worker-manager separated Appserver cluster -

Demo/Hands-on● Deployment Automation● Server Monitoring● Production Deployment Guidelines● Common issues in Production Setups

WSO2 Product deployment patterns

● Standalone

● Clustered

● Cloud based

Standalone Deployment

● User engages with one or many independent nodes

Clustered Deployment

● User engages with the load balancer● Highly available● Concept of worker and manager nodes

Worker-Manager Separation ?

➔ Proper separation of concerns◆ worker node - can ONLY deploy artifacts and read

configurations◆ manager node - authorized to add new artifacts and

make configuration changes➔ Lower memory foot in the worker nodes

◆ lesser OSGi bundles➔ Improved security

◆ management nodes can be behind the internal firewall & be exposed to clients running within the organization only, while worker nodes can be exposed to external clients.

Cloud Based Deployment

● Extension of the Clustered deployment

● Highly scalable

● Span across multiple IaaSs

Why Deployment synchronization ?

• Artifact distribution and deployment should be transparent to the end-users

• Manual artifact copying is acceptable for some extent in standalone product clusters.

• Will that work in an elastically scaling cloud environment ??

Deployment Synchronization

● Distribute deployment artifacts and related meta-data across the cluster

● Make the cluster nodes consistent● Automated synchronization based on SVN ● Management nodes commits and worker nodes check-out

Deployment Synchronization<--Manager Node --><DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>true</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId></DeploymentSynchronizer>

<-- Worker Node --><DeploymentSynchronizer> <Enabled>true</Enabled> <AutoCommit>false</AutoCommit> <AutoCheckout>true</AutoCheckout> <RepositoryType>svn</RepositoryType> <SvnUrl>http://10.100.3.115/svn/repos/as</SvnUrl> <SvnUser>wso2</SvnUser> <SvnPassword>wso2123</SvnPassword> <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId></DeploymentSynchronizer>

Registry Mounting

● An approach to share registry space across cluster nodes● Local Data repository

○ Should not be shared● Configuration registry

○ Shared across multiple nodes of the same product● Governance registry

○ Shared across product platform

Registry Mounting

Worker-Manager Separated Cluster - Demo/Hands-On

● Setting up user management and registry databases● Configuring the Elastic loadbalancer● Configuring WSO2 AS as a management node● Configuring WSO2 AS as a worker node

Deployment Automation

Configuration Management

No matter how small you are

adapt yourself to configuration management

Abstracted Configurationnode001 { appserver { version => "5.2.1", target => "/opt/appserver", cluster_domain => "mgt", }}

node002 { elb { version => "2.1.0", offset => "0", target => "/opt/elb", user => "root", } }

Patch Management

Orchestration

Powered by Puppet

https://github.com/wso2/Puppet-Modules

Server and Health Monitoring

● JMX based monitoring○ JConsole, Graphite

● Message Tracing Tools○ Wireshark, tcpmon

● System monitoring tools○ Ganglia, Cacti, Nagios

● Alerting○ Email, SMS, Phone-call alerts

Production Deployment Guidelines

● Changing the default administrator credentials● Replacing SSL certificates/keys● Hardening the Operating System● Turning off unused services/ports● Switching the default H2 registry database to a production

ready database● Using secure vault to encrypt passwords in configuration files● Tune database connection pools

Production Deployment Guidelines contd..

● Disabling HTTP access logs● Tuning heap memory● Tuning GC● Running servers as background processes

○ nohup, YAJSW

Patching

● Follow the recommended procedure● Automate the patch applying process● Test the patch in dev and staging environments

○ Automated process is a must● Update production servers and restart in a RR manner● Maintain a log for patches

Common errors in production

● IO errors - Connection reset by peer, Broken pipe, Too many open files, I/O reactor has been shut down

● Database connection pool errors● OutOfMemoryErrors

Production Troubleshooting Tips

● carbondump○ WSO2_HOME/bin/carbondump.{sh,bat}

● Replicate the issue in staging● Report JIRAs with detailed information

○ Upload carbondump output to the provided FTP server○ Attach the relevant deployment artifacts and logs

Sanjaya Ratnaweera

Blog: http://samudura.org

Follow: @sanjayar

Amani Soysa

Blog: http://sparkletechthoughts.blogspot.com/

Follow: @poohdedoo

Thank You