Load Runner

Performance Testing Using Load Runner

TABLE OF CONTENTS

1

Load Runner

Introduction 5

Definition for Performance testing 5

Types of Performance Testing 5

Need for Performance testing 5

Automate Performance Testing 6

Automated Performance Testing Phases 6

Requirements collection 7

Design 7

Build 7

Execute 8

Analysis and Tuning 8

Performance Test Planning 10

Load / Performance Tool Evaluation & Selection 10

Test Process / Methodology and Test Strategy 11

Performance Test Start/Stop Criteria 11

Test Environmental Setup and Pre-Requisites 12

Analysis and Report Generation 12

Pass / Fail / Exit Criteria 12

Common Implementation Issues

13

Preparation 13

Test Environment Setup 13

Customization 14

Execution/Analysis 15

Introduction to Load Runner 16

Load Runner Terminology 17

Load Runner Vuser Technology 17

Creating Scripts Using Load Runner 18

Recording Vuser Scripts using VuGen 18

Enhancing the scripts 19

Configure Run time Settings 20

Executing Scenario using Controller 29

2

Load Runner

Scheduling a Scenario 29

Setting Up Load Generator Machines 30

Configuring Load Generators 30

Setting up the Monitors 31

Analyzing Results 33

Summary Report 34

Running Vusers Graph 35

Hits per Second 36

Transaction Summary 37

Average Transaction Response Time 38

Tuning 39

Information Captured for Tuning 39

Performance Counters to be captured 39

Abstract

3

Load Runner

The incredible pace of change and the explosion of software complexity introduce

tremendous risk into software development process. Rigorous testing is the most

common strategy to quantify and reduce this risk to the business. The question for

developers, QA teams, and management alike is how accurately and thoroughly they

can validate the system’s performance before Go Live – without breaking the Budget.

By conducting performance testing, an organization can verify that the application(s)

meet the Business needs before Go Live. This ensures that there are no Deployment

Risks, and Quantifies the impact of changes on End-User’s experience by pinpointing

the Failed Components for Rapid Resolution.

This paper presents a complete practical approach for performance testing and

covers the best approach which includes the following:

What is performance testing?

Why should you automate it?

What is the right process for performance testing?

Who should be involved in a good performance test?

What are the requirements need to be collected for performance Testing?

How to plan for performance Testing?

What are the common implementation issues faced during the performance

testing?

How to conduct a performance testing using Load Runner?

o Creation of Scripts

o Enhancing Scripts

o Creation and Execution of Scenarios

o Analyzing Results

o Tuning the application.

Introduction

What is Performance Testing?

Performance Testing is aimed at verifying the system’s performance requirements

like response time, Transactional throughput and no of concurrent users.

Performance testing is used to accurately measure the End-to-End performance of a

4

Load Runner

system prior to Go Live. It will identify the loop holes in Architectural Design which

helps to tune the application.

It includes the following:

Emulating ‘n’ number of users interacting with the system using minimal

hardware.

Measuring End-User’s Response time.

Repeating the load consistently

Monitoring the system components under controlled load.

Providing robust analysis and reporting engines.

Types of Performance Testing

Load Testing:

To verify that system can handle the expected load, on deploying it under Real Time

conditions.

Stress Testing:

To identify the application’s Break Point, also to measures whether the application’s

environment is configured to handle the expected or potentially unexpected high

transactional volume.

Scalability testing:

Scalability testing integrates very well with performance testing. The purpose of

scalability testing is to determine whether the application automatically scales to

meet the growing user load.

Volume testing:

To verify the stability of the system with respect to handling large amounts of data

over extended period of time.

Need for Performance testing

To verify the below mentioned:

Application’s response for the intended number of users

Application’s maximum load resisting capacity

Application’s capacity of handling the number of transactions required by the

Business

Application’s stability under expected and unexpected user load

5

Load Runner

Ensuring that users have proper response time on production

Automate Performance Testing

Automating Performance Test is a discipline that leverages people, process, and

technology to reduce the Risks of the Application. It is about applying production

workloads to pre-deployment systems while simultaneously measuring system

performance and End-User experience.

Below are the loopholes in manual Load Testing that leads to Automation.

Difficult to measure the performance of the application accurately.

Achieving the synchronization among the users.

‘n’ number of real time users are required to involve in Performance Testing

Difficult to analyze and identify the results & bottlenecks.

Increases the infrastructure cost.

Automated Performance Testing Phases

Requirements Collection

Design

This is the primary phase where team will be gathering the requirements of the

performance testing. Requirements can be Business, Technical, System and Team

requirements.

Business requirements includes

1. An Application Overview: Performance Team should have good

understanding of the Application prior to Testing

6

Planning/Design

Build

Execution

Analysis and Tuning

Load Runner

2. Business Processes List: Team should gather complete information about

list of key Business activities that will be performed by an End User on the

system.

3. Business Process Flows: Document, which contains step-by-step details of

the Business processes identified by Business Team.

4. Transaction List: List of Key activities with in the Business process that

need to be performed in unit time.

Technical requirements should be collected from the System Administrators and

Database Administrators, which include the following:

1. Complete Architecture of the Environment

2. How many no. of users that the system should support during normal and

peak hours.

3. How many no. of transactions per unit time should be processed by the

system

4. Response time for all the Business Processes.

5. What are the different system workloads that will be experienced in

production

6. How the load balancing will be done during the testing

Build

This phase consists of automating the requirements collected during the design

phase. It includes

1. Scripting: Record the business processes gathered in the design phase into

Automated scripts.

2. Transactions: These are the points that measure the performance of a

particular key activity with in the business process.

3. Parameterization: Input data needs to be passed for the scripts for

performing the Business activity on an application.

4. Scenarios: The complete End-to-End Business Workflow, which simulates the

Real Time Environment.

7

Load Runner

5. Monitors: Decide which servers or machines need to be monitored.

Environment setup consists of assembling Hardware, Software and Data required for

executing a successful real load. This may involve working with the systems, DBA,

Operations and Business Teams.

Execute

Execution will be done in multiple phases. It consists of various types of testing like

Baseline Testing

This testing is done to ensure that both applications’ functionality & environment

setup for performance testing is proper. This will be done with 10 to 20 users.

Performance Tests

This includes running the scripts with Average and Maximum load with specified no of

transactions. We will be simulating the real time environment with different load

conditions using different Scenarios, Think Time, Pace Time and no. of users.

Benchmark Tests

These are designed to measure and compare the performance of each machine type,

environment or build of the application in ideal situation. These tests are run after

the system undergoes scalability testing to understand the performance impact of

different architectures.

Analysis and Tuning

During the performance testing we will be capturing all the details related to the

system like Response time and System Resources for identifying the major

bottlenecks of the system. After the bottlenecks are identified we have to tune the

system to improve the overall performance.

Performance Optimization

This quantifies to improve the performance of the system as measured by improved

end-user response time or by reducing the overall required hardware infrastructure.

Performance Test Planning

To make any operation successful, planning plays a major role. According to the

80-20 theories, 80% of the time should be spent in planning and 20% in real time

8

Load Runner

plan execution/implementation. In the similar procedure, software performance

test planning is very crucial as well.

Any Software Performance Test Plan should have the minimum contents as

mentioned below,

Performance Test Strategy and scope definitions.

Test process and methodologies to follow in different test phases.

Test tool details (Tool Evaluation, Selection, Configuration and Add-ins)

Test cases details including scripting, library development and script

maintenance mechanisms to be used in every performance Test Phase.

Resource allocations and responsibilities for Testers.

Risk management definitions.

Test Start /Stop criteria along with Pass/Fail criteria definitions.

Test environment setup requirements. (Hardware Setup, Software Setup,

Os Configurations, Third Party Utilities Setup etc.)

Virtual Users, Load (Iterations Vs Users), Volume Load Definitions for

Different Load/Performance Test Phases.

Results Analysis and Reporting format definitions.

Load / Performance Tool Evaluation & Selection

Evaluation of tool is an important task in performance test automation where

there are several parameters to be considered. In case the tool evaluation and

selection is completely done by the client then we don’t need to consider this.

While selecting any tool for Load or/and Performance testing the following things

should be analyzed:

Test budget Vs Available Load /Performance tools in the market mapping

to the Budget.

Protocols, Development & Deployment Technologies, Platforms, Middle-

Ware tools / Servers, Third-Party Integrations with the application under

test Vs support for these factors in the available tools with prioritization of

the availability in the tool for the scope of expected test.

9

Load Runner

Complexity of the tool usage Vs Availability of the tool experts along with

the timeline requirements for the tool scripting / load scenario creation /

tool configuration with respect to man-hours and other resource

requirements.

Tool limitations and work-around factors mapped with the current scope of

testing.

Tool’s integration / Portability factors with other tools used for monitoring

and Analyzing.

On evaluation and selection of the base tool, third party monitors / tools to

be used in integration with main Load testing tool should be defined.

Test Process / Methodology and Test Strategy

Test process, methodology and strategy for any Performance testing will

definitely vary from project to project based on the client’s requirement /

company. By using a methodology a project can make sure that resources are

applied to the problem effectively and the people involved in the work in a

structured way.

Performance Test Start/Stop Criteria

In performance testing, there should be a defined Start and Stop criteria to avoid

the ad-hoc process resulting in weird outputs. This Start / Stop criteria should be

well defined in the performance Test Plan along with all the risk management

factors.

The start of any Performance testing should be done for any application or any

module of the application during the design, database structure and application’s

development platform is finalized with no major changes at any layers of the

application that effect directly or indirectly.

Once the Performance test is conducted, results are collected and analyzed. The

cycles of tuning will be carried out but definitely the end criteria should be well

defined to stop this Performance testing. Tuning will be done based on various

factors like product release timelines (Project Deadlines) and client requirements

in terms of system stability, scalability and reliability under load and performance

tolerance criterions.

10

Load Runner

In some projects the end criteria is defined based on the client performance

requirements defined for each section of the application. So, if the product

reaches to the expected level then that can be considered as the end criteria for

performance testing.

Test Environmental Setup and Pre-Requisites

In performance testing, test environment is composed of hardware environment

like Machine, Memory, Virtual Memory, Number of CPU’s etc., software

environment like OS, File system settings, free space for logs and application

configuration along with middle tier server and associated configurations,

Database, Network etc. his setup also involves the Automation Tools installation,

configurations, Agent Installations, setup in Load Clients, Third Party Monitors on

Clients / Servers etc.

Analysis and Report Generation

Once the results are ready, analysis is the most important job in the whole

performance test. The main criteria for analysis should be to find out the

bottlenecks and to track out the transactions / users / hits that create the highest

level of performance reduction.

The main factors to be considered are:

Request / Response Timings from client to server

Transaction Timings Vs Users

Transaction Timings Vs Load

Transaction Timings Vs CPU Utilization

Server Memory Utilization

Database server performance using DB server monitors

Web / Application server performance

Paging Rate Ratios

Load Balancer Factors

After consolidating all the results, the statistical analysis should be done and the

performance test report should be created such that the development, design-

architecture, database group can get a clear idea about the application’s

performance and all the other factors affecting it. After analysis, based on the

scope tuning like OS tuning, Database Tuning, Memory Up-gradations, CPU counts

in Multi-CPU machine, Code tuning, Application Server cache tuning or other

11

Load Runner

tunings can be done. Same tests executed earlier should be repeated in the

tuned environment to make sure that there are differences in the results before

and after tuning. These tuning cycles can be repeated until the required

performance is achieved.

Pass / Fail / Exit Criteria

At the end of the Performance test execution, we can find the number of failed

users, failed transactions or failed scripts. But the Pass / Fail criteria for a

particular test should be well defined before execution of the test, as this criterion

entirely depends on the scope of testing and the environment. The failed items

may be due to any of the performance test parameters. So the failed item details

should be included in the application’s limitations list. By doing so, the limitations

list can give a clear picture about the performance limitations with respect to load

parameters which in turn will give inputs for the tuning / enhancements for the

application. Based on the test results, tuning cycles will be repeated but the exit

criteria should be pre-defined to avoid unnecessary tuning life cycles.

Common Implementation Issues

This section describes the most common issues that occur during the load testing

and also suggests strategies for addressing those issues.

Preparation

Planning:

The biggest issue with planning is that it is not performed adequately. A good

process is to allow three months for a load test with completion at least a month

before “go live”. Preparation, development, execution/analysis and results

summary each should be allocated one month. Typically the results summary can

be performed in parallel with late-term project tasks.

Analysis and Design:

The first issue to be considered is “what should I test?” and the related question

“what activity does my user community create”. In many ways this is black art of

testing. Often the vendors, system architects, IT and business managers make

educated guesses about the “representative user load profile”. This process

involves a study of the system’s operation profile. This also includes an interview-

analyze-negotiate process with user community.

12

Load Runner

Business processes:

High volume business processes / transactions should be built into the test.

Choosing too few might leave gaps in the test while choosing many will expand

the script creation time. We can effectively model the most common 80 percent

of the transaction throughput but trying to achieve greater accuracy is difficult

and expensive.

Test Environment Setup

Load testing requires a complete, stable and independent environment for testing.

Hardware:

To get the reliable results, the physical components of the test environment must

represent the production environment. This offers the advantage for ongoing

testing. This is particularly useful for maintenance or system expansion.

Software:

In addition to the hardware required for load test, the test bed must also have

fully installed and functioning software. Since Load Runner (Vuser) represents a

real time user (s), the system should successfully support all the user actions.

Network:

Organization’s network infrastructure shared by computers may create significant

amount of “background noise”. While it is probably impossible to accurately

model each and every network access, but it is good to examine the current

network utilization and understand the impact on network traffic.

Geography:

Often the system under test will support the global enterprise. In this

environment tests may often need to be run at remote sites across the WAN.

WAN connectivity need to be emulated in the lab or assumptions must be made.

Customization

Customization of Load Runner is possible and has been performed in many

environments. It will take considerable engineering effort and is not an out-of-

box-solution. Specifically customization of load runner requires:

1. Load Runner expert familiar with Template programming

2. System specialist familiar with the application’s API and administration.

13

Load Runner

3. An additional one-month of project time to investigate and construct a custom

Load Runner drive.

Functional Support:

One of the most important factors in script creation productivity is the amount of

functional support provided – access to individuals who understand the

application’s functionality. When a test team member encounters a functional

error while scripting, the business Operation doesn’t function properly. The team

member typically has to stop since he or she is not equipped with the skill to

solve the issue. At that point, script creation is temporarily halted until a

functional team member helps to resolve the issue.

Test design Quality:

The second factor of script development is the quality of the test design. Ideally

the test design should specify enough information for an individual with little or

no experience to test builds. System test documentation is an excellent source of

this information. Often designs are incorrect or incomplete as a result, any

omission will require functional support to complete the script development.

Business Process Stability:

To load/ stress a large system, at first the system’s business processes should

function properly. It is typically not effective to attempt a load test a system that

doesn’t even work for the user. This means that the system needs to be nearly

completed.

System Changes:

The last factor in script development is the frequency of system changes. For

each system revision, test scripts must be evaluated. Each test should be

unaffected, require simple rework, or complete reconstruction. While testing tools

are used to minimize the effect of the system change, which will reduce the

scripting time.

Test Data Load:

The system should be loaded with development test data. This data often comes

from a legacy conversion and predecessor to the volume data for the test.

14

Load Runner

Execution/Analysis

System Changes:

Often there is shift between the systems used for development and execution

analysis. The first step is to execute all the business scenarios with single user.

This typically involves working with each test to account for system changes like

difference in data load, configuration, system version etc which may cause issues.

It is recommended to perform a backup of the development environment and/or

database that may be restored for execution.

Volume data:

System performance may vary widely with the volume of the system data. The

system under test should have sufficient data to model the size of the product

database at the time of deployment. It is also good to expand the database size

to model the system performance with more than expected data for one or two

years.

System Support:

Just as the development requires functional support, execution/analysis requires

system support because the test team may not understand the system

architecture and technology. The purpose of the system support is to help in

interpreting the load runner results and performance log files. While Load Runner

will describe the facts based on which the system support will describe the reason

and suggest remedies for the problems. These suggestions can be implemented

and the tests will be rerun to cross verify the new implementation.

Introduction to Load Runner

Mercury’s Load Runner is an efficient & effective tool used to conduct

performance testing. The following reasons prove its efficiency:

1. On - demand Production Work Loads: Load Runner is able to drive ‘n’

number of virtual users executing business process to emulate the real time

conditions.

15

Load Runner

2. Enterprise Environment Support: Load Runner supports nearly 40

protocols; this includes Web, J2EE, .NET, XML, SAP, Siebel, Oracle, PeopleSoft,

wireless, Citrix, and other client-server applications.

3. Enterprise Monitoring Support: Load Runner’s real-time performance

monitors provide detailed metrics on all parts of the system under test. This

includes Web servers, application servers, databases, ERP and CRM systems,

firewalls, and load balancers. Load Runner can identify hardware limitations

and software configuration issues that might otherwise go undetected.

4. Diagnostics: Load Runner can be used to trace, time and troubleshoots

individual application components under load. Users can now drill down from

a slow end-user transaction to the bottlenecked method or SQL statement

that is causing the slowdown. This granularity of results data ensures that

every load test provides development with actionable results, thus reducing

the cost and time required to optimize J2EE, Siebel, and Oracle deployments.

5. Infrastructure Tuning: Load Runner will provide a customizable

methodology to isolate and resolve infrastructure bottlenecks in the system

prior to going into production. Over 70 percent of performance bottlenecks

are due to system infrastructure issues. By detecting and resolving these in

pre-production, Load Runner saves valuable time and money.

6. Automatic Analysis: Load Runner’s Auto-Correlation wizard automatically

takes all the monitoring & diagnostics data and provides the top-five causes of

performance degradation. As a result, Load Runner has the industry’s deepest

and broadest analysis. Converting performance-testing results into actionable,

precise data to development dramatically reduces time to resolution and

allows for more testing cycles. This is a critical piece in ensuring that a high-

quality application goes into production.

7. Ease of Use: Load Runner is built from the ground up for QA users. It

provides a visual scripting language, Data and autocorrelation wizards and

Active Screen technology to make scripting and running of performance tests

as easy as possible.

16

Load Runner

Load Runner Terminology

Scenario: A scenario defines the events that occur during each testing session.

Scenario defines and controls the number of users to emulate, the actions that they

perform and the machines on which they run their emulations.

Vusers: In the scenario, Load Runner replaces human users with virtual users or

Vusers. When you run a scenario, Vusers emulate the actions of human users

working with your application.

Vuser Scripts: The actions that a Vuser performs during the scenario are described

in a Vuser script. When you run a scenario, each Vuser executes a Vuser script.

Transactions: To measure the performance of the server, you define transactions.

A transaction represents an action or a set of actions that you are interested in

measuring.

Controller: LoadRunner Controller will be used to manage and maintain your

scenarios. Using the Controller, you control all the Vusers in a scenario from a single

workstation.

Load generator: When you execute a scenario, the Load Runner Controller

distributes each Vuser in the scenario to a load generator. The load generator is the

machine that executes the Vuser script, enabling the Vuser to emulate the actions of

a human user.

Load Runner Vuser Technology

17

Controller

Remote agent Dispatcher

Agent

Vuser

Vuser

Vuser

Load Runner

Load generator

Creating Scripts using Load Runner

We have to create the scripts by recording the steps specified in the business

scenario document. Creating the script consists of the flowing steps.

Recording Vuser Scripts using VuGen

We can use VuGen to develop the Vuser script by recording user performing typical

business processes on a client application. Each Vuser script that you create with

VuGen can directly communicate with a server by executing calls to the server API

without relying on client software. In addition, when a user communicates directly

with the server, system resources are not used on a user interface. This lets you run

large number of Vusers on a single workstation.

1. Open the Virtual User Generator.

18

Record Vuser Script

Enhance Vuser Script

Configure Runtime Settings

Run the Vuser Script in Stand Alone Mode

Load Runner

2. Select the Protocol for recording.

3. Click on Start recording.

4. Enter URL/Path of the application for recording.

5. Record all the steps for the specified business scenario

Enhancing the scripts

While recording a script or after recording, you can enhance its capabilities by adding

the following functions to the script.

1. General functions

2. Protocol specific functions.

3. Standard ANSI C functions.

4. Parameterization.

5. Correlation.

General Vuser functions:

General Vuser functions greatly enhance the script functionality of any Vuser script.

You can use general Vuser functions to measure the server performance, control

server load and debugging code or retrieve run-time information about the Vusers

participating in the scenario.

Protocol Specific Functions:

There are several library functions that you can use to enhance your script. For

example LRS functions for Win Socket applications, LRD functions for database

applications etc.

Standard ANSI C Functions:

19

Load Runner

You can enhance your Vuser scripts by adding ANSI C functions to the script. ANSI C

functions allow you to add comments, Control flow statements and conditional

statements so on to Vuser scripts.

Parameterization:

When you record a business process VuGen generates a script that contains actual

values used for recording. Suppose you want to perform script actions like query,

Submit using different values from those recorded values. To do this you need to

replace the recorded values with parameters. This is called as parameterization.

Correlation:

Correlation is used to capture the dynamic value for a particular field at run time.

The primary reason for correlating statement is

1. To simplify or optimize your code

2. For Dynamic Data

3. To accommodate unique records.

You can also enhance the scripts by

1. Inserting transactions into a Vuser script.

2. Inserting Rendezvous point into a Vuser script

3. Inserting comments into the script.

4. Sending messages output.

5. Handling errors in Vuser Scripts.

Configuring Run time Settings

The run time settings will define the way that the script runs. These settings are

stored in the file default.cfg located in the Vuser directory. Runtime settings are

applied to Vusers when you run the scripts in VuGen or controller. We have to adjust

20

Load Runner

the run time settings based on the no. of transactions/hour. We will be calculating

this using the formula.

Transactions / hour = no. of user* 60 /

Action section Execution time + Think time + Pacing Time)

Run Logic

Number of Iterations: Based on the number of iterations set, Load Runner repeats

all the Actions for specified number of times. If you specify scenario duration in the

Controller’s Scheduling settings, the duration setting overrides the Vuser iteration

settings. This means that if the duration is set to one hour, the Vusers will continue to

run as many iterations as possible in one hour, even if the run-time settings set to

one iteration.

Pacing

21

Load Runner

The Pacing Run-Time settings let you control the time between iterations. The pace

tells the Vuser how long to wait between iterations of your actions.

You instruct the Vusers to start each iteration using one of the following

As soon as the previous iteration ends: The new iteration begins as soon as the

previous iteration ends.

After the previous iteration ends with a fixed or random delay of …: Starts

each new iteration, with a delay of specified amount of time after the end of the

previous iteration. Specify either an exact number of seconds or a range of time.

At fixed or random intervals, every … [to …] seconds: You can specify the time

between iteration either a fixed number of seconds / range of seconds. For example,

you can specify to begin a new iteration every 30 seconds, or at a random rate

22

Load Runner

ranging from 30 to 45 seconds from the beginning of the previous iteration. Each

scheduled iteration will only begin when the previous iteration is complete. The

actual amount of time that the Vuser waits between the end of one iteration and the

start of the next one appears in the Execution Log when you run the script.

Log

Enable Logging

This option enables automatic logging during replay. VuGen writes log messages that

you can view in the Execution log. This option only affects automatic logging and log

messages issued through lr_log_message. Messages sent manually, using

lr_message, lr_output_message, and lr_error_message, are still issued. The Log

run-time settings allow you to adjust the logging level depending on your

development stage.

Standard Log

Creates a standard log of functions and messages sent during script execution to use

for debugging. Disable this option for large load testing scenarios. If the logging level

23

Load Runner

is set to Standard, the logging mode is automatically set to JIT logging when adding it

to a scenario.

Extended Log

Creates an extended log, including warnings and other messages. Disable this option

for large load testing scenarios. If logging is disabled or if the level is set to Extended,

adding it to a scenario does not affect the log settings. You can specify additional

information, which should be added to the extended log using the Extended log

options:

Parameter substitution

Select this option to log all parameters assigned to the script along with their values

Data returned by server

Select this option to log all of the data returned by the server.

Advanced trace

Select this option to log all of the functions and messages sent by the Vuser during

the session. This option is useful when you debug a Vuser script.

Think Time

24

Load Runner

Vuser think time emulates the time that a real user waits between actions. For

example, when a user receives data from a server, the user may wait several

seconds to review the data before responding. This delay is known as the think time.

VuGen uses lr_think_time functions to record think time values into your Vuser

scripts.

By default, when you run a Vuser script, the Vuser uses the think time values that

were recorded into the script during the recording session. VuGen allows you to use

the recorded think time, ignore it, or use a value related to the recorded time.

Ignore think time: Ignore the recorded think time option replay the script by

ignoring all lr_think_time functions.

Replay the think time: The second set of think time options let you use the

recorded think time.

25

Load Runner

As recorded: During replay, use the argument that appears in the

lr_think_time function. For example, lr_think_time (10) waits ten seconds.

Multiply recorded think time by: During replay, use a multiple of the

recorded think time. This can increase or decrease the think time applied

during playback

Use random percentage of the recorded think time: Use a random

percentage of the recorded think time. You set a range for the think time

value by specifying a range for the think time.

Limit think time to: Limit the think time’s maximum value to the specified

value.

Miscellaneous Run-Time Settings

Error Handling

You can specify how a Vuser handles errors during script execution. By default, when

a Vuser detects an error, it exits. You can use the run-time settings to instruct a

Vuser to continue script execution when an error occurs. To do so, select the

Continue on Error check box in the Miscellaneous run-time settings.

You can also instruct VuGen to mark all transactions in which an lr_error_message

function was issued, as Failed. The lr_error_message function is issued through a

programmed If statement, when a certain condition is met.

26

Load Runner

Multithreading

Vusers support multithread environments. The primary advantage of a multithread

environment is the ability to run more Vusers per load generator. Only thread safe

protocols should be run as threads.

To enable multithreading, click Run Vuser as a thread.

To disable multithreading and run each Vuser as a separate process, click

Run Vuser as a process.

Automatic Transactions

You can instruct Load Runner to handle every step or action in a Vuser script as a

transaction. This is called using automatic transactions. Load Runner assigns the step

or action name as the name of the transaction. By default, automatic transactions

per action are enabled.

To disable automatic transactions per action, clear the Define each action

as a transaction check box. (enabled by default)

To enable automatic transactions per step, check the Define each step as a

transaction check box. (disabled by default)

27

Load Runner

Speed Simulation

Using the Speed Simulation settings, you can select a bandwidth that best emulates

the environment under test. The following options are available:

Use maximum bandwidth: By default, bandwidth emulation is disabled and

the Vusers run at the maximum bandwidth that is available over the network.

Use bandwidth: Indicate a specific bandwidth level for your Vuser to

emulate. You can select a speed ranging from 14.4 to 512 Kbps, emulating

analog modems, ISDN, or DSL.

Use custom bandwidth: Indicate a bandwidth limit for your Vuser to

emulate. Specify the bandwidth in bits, where 1 Kilobit=1024 bits.

Proxy

Proxy runtime settings are used to provide the Proxy server details to the scripts.

28

Load Runner

No Proxy: For all the Vusers make the connection directly to the internet, without

using Proxy Server.

Obtain the Proxy Server settings from the default Browser: All Vusers use the

proxy settings of the browser on the machine they are running. This is the default

option.

User Custom Proxy: All Vusers connect to the Internet through a Proxy using the

settings you specify.

Use Proxy: Enabling this check box provides the Address of the proxy to use and

port. If the proxy requires authentication information provide this in the proxy

authentication window specified below.

29

Load Runner

Executing Scenario Using Controller

Scenario: A scenario defines the events that occur during each testing session.

Scenario defines and controls the number of users to emulate, the actions that they

perform and the machines on which they run their emulations.

Manual Scenario: Select this method if you want to build a manual scenario. You

build a manual scenario by creating groups and specifying the script, the load

generator, and the number of Vusers included in each group.

Use the Percentage Mode to distribute the Vusers among the scripts:

Select this option if you want to build a manual scenario by specifying a

number of Vusers to be distributed among the selected Vuser scripts.

Goal Oriented Scenario: Select this method to have Load Runner build a scenario

for you. In a goal-oriented scenario, you define the goals you want your test to

achieve, and Load Runner automatically builds a scenario for you, based on these

goals. This scenario is mainly used for stress testing.

Scheduling a Scenario

Ramp up: Specifies the entry criteria for the users into the server.

Load All users Simultaneously: Will load all the users at a time after the

scenario is started.

Start {No. of users} after every {Time}: will load the specified number

of users in to the server in a given time.

Duration:

Run until Completion: Will run the scripts once. If the number of iterations

specified inside the Runtime Settings, Script will be executed for specified

number of iterations for each user.

Run for {Time} after ramp up has been completed: Scripts will be run

for specified duration after all the users were ramped up. Script will be

repeated for ‘n’ number of times based on the duration specified. On selecting

this option, Run Logic runtime setting specified in the script will be

overwritten.

Run Indefinitely: Scripts will be running indefinitely till we stop the scenario.

This option will be used for conducting the Volume testing of an application.

30

Load Runner

Ramp Down: Specifies the exit criteria for the users from the server.

Stop all Vusers Simultaneously: Exits all the users from the server at a

time.

Stop {No. of users} after every {Time}: Stops a certain number of

Vusers within a specified time frame.

Setting Up Load Generator Machines

After installing the load runner agent software on the Load Generator machine we

have to configure the Agent for connectivity with the controller machine. Follow the

below steps for configuring the Agent

1. Go to Programs -> Load Runner ->Advance Settings -> Agent Configuration. On

selecting this it will display the following window

Select the Enable Firewall Agent option and click on Settings button. It will

display all the setting related to the agent configuration. Set the following properties

for configuring the agent.

Local Machine Key: Specify the IP Address of the controller machine to establish a

unique connection between the Controller the agent machine.

Configuring Load Generators

For configuring the load Generator click on the Generator button in Controller

design window and add the load generator machine by specifying the name of the

machine. Click on Connect button to check whether the connectivity between the

controller machine and Load generator machine can be established. If the

connectivity is failed please check the following.

1. Login into the Load Generator machine ensure that Agent process is launched

and running.

31

Load Runner

2. Check the network connectivity between controller machines by running the

command Ping machine_name at command prompt of the controller/ agent

machine.

3. Verify the Load Generator Machine name / IP address in the load generator

window.

4. Verify whether the agent machine configured using Agent configuration.

5. Verify whether any firewall exists.

Setting up the Monitors

Monitors

Load Runner uses a suite of integrated performance monitors to quickly isolate

system bottlenecks with minimal impact on the system. The suite consists of

monitors for the network, application servers, Web servers and database servers.

These monitors are designed to accurately measure the performance of every single

tier, server and component of the system during the load test. By correlating this

performance data with end-user loads and response times, IT groups can easily

determine the source of bottlenecks. In addition, they can collect and manage all

system performance data from the Load Runner Controller. We can configure the

monitors in controller to capture these details.

Configuring Load Runner Monitors.

1. Go to Run tab of controller window.

2. Select the corresponding monitor (Web / Application / Database server) from

Available Graphs section

3. Drag and drop the monitor into any of the graphs.

4. Right click on the graph and select Add Measurements

Now it will display the following window.

32

Load Runner

5. Click on Add(top) and enter the machine name of the Web/App/DB server

6. Now click on the Add (Down) button and select the required Performance

counters to be measured during the scenario execution.

7. Click on OK button.

If controller is not able to capture the details after configuring please ensure the

following.

1. Verify the connectivity between the Load Runner Controller machine and

Server Machine.

2. Verify whether you are Admin user or you have sufficient privileges to capture

the performance counters from the server machine.

Analyzing Results

Once the results are ready, analysis is the most important job in the whole

performance test if the requirements of performance test are not defined. The main

criteria for analysis should be to find out the bottlenecks and we should be able to

track the transactions / users / hits that create the highest level of performance

33

Load Runner

reduction. Based on the test run, Load Runner will generate various graphs and

reports. This phase contains:

Analysis

Once the load test is completed user can correlate the various graphs to analyze the

results and to identify the business scenarios causing high CPU utilization and

network traffic.

Monitoring

It includes observation of what is happening in each tier of the application, web

server, Application server and Database server. We can analyze the system

resources utilized and network delays.

Tuning

It includes observation and understanding of the particular query or activity which

are not meeting the response time and tuning it by creating indexes, or reordering

the queries and by modifying the hardware( if required).

Diagnostics

Performance testers have a single, unified view of how individual tiers, components

and SQL statements impact the overall performance of a business process under load

conditions. With diagnostics, performance testers using Load Runner can see all

components touched by an end – user transaction. More over, users can also see

how long each component takes and how many times it is called. With this

information corresponding team can focus on resources to improve the end-user

experience by targeting the most significant area of the web application and

database server bottlenecks.

Summary Report

The summary Report provides the information about the scenario execution. It

provides the information like Running Vusers, Throughput, Hits Per second, HTTP

Responses per Second, Transaction Summary and Average Transaction Response

time. 90 percent column indicates the maximum response time for ninety percent of

the Transactions.

34

Load Runner 35

Load Runner

Running Vusers Graph

Running Vusers graph displays the number of users that executed the scripts and

their status during the execution. This graph is useful to determine the load on the

server at any given moment.

Hits per Second

The Hits per Second graph shows the number of HTTP requests made by Vusers to

the Web servers during the scenario execution. This graph helps you to evaluate the

amount of load Vusers generate in terms of number of hits.

36

Load Runner 37

Load Runner

Transaction Summary

Transaction summary graph summarizes the number of transactions in the scenario

that failed, Passed, stopped and ended with errors.

38

Load Runner

Average Transaction Response Time

This graph displays the average time taken to perform transactions during each

second of the scenario run.

39

Load Runner

Tuning

Information Captured for Tuning

For analyzing the results and for identifying the bottlenecks capture the following

information during the Scenario execution.

1. Run the SQL Server Trace file during the Scenario Run.

2. Capture the Performance counters during the run into the Load Runner

controller (if you have access to LR Monitors) otherwise use the windows

performance monitor to capture the same.

3. Run the Diagnosis tools, if you find any Hardware issues.

Performance Counters to be captured

1. CPU Utilization

2. Avg. Disk Queue Length

3. Network Interface

4. System

5. Server

6. Process

7. Physical Disk

8. Logical Disk

9. Memory

Few Tips on Resources utilization of the Servers

Parameter Tips

Utilization

None of the CPUs should have utilization of more than

80 percent.

Avg. Disk Queue Length Should not be more than 2.

Load Average More than three processes should not wait for the CPU.

CPU Intensive Processes

Any non-DB, system processes utilizing CPU intensively

warrant attention.

Swapping Minimal to no swapped-out processes

Paging Minimize page faults

Data Buffer Hit Ratio More than 70 percent

High Disk Activity

Disk should have less than 50-70 I/Os per second. Also,

the disks should be less than 60 percent busy.

40

Load Runner

Long Disk Queues

Minimal to no disk request queue. A queue of 2-4

warrants attention.

Un-indexed Tables All the tables should have at least one Primary Key.

Measurement Calculations:

The less time per transaction, the greater a system's throughput, as expressed by

the formula derived from Little's Law:

Server Thruput = # of real simultaneous users / ( Average Response Time + Think

Time )

So when no think time is used, the number of virtual users can be calculated by this

formula suggested by Menasce & Almeida:

# of real simultaneous users * [ Response Time / ( Response Time + Think Time ) ]

For example, to simulate 1,000 real users using 2 second response time and 20

second think times:

1,000 x [ 2 / ( 2 + 20 ) ] = 91

System Load

is measured by a rate -- the total number of tasks performed during a certain

period (an hour, day, week, month, quarter, year, etc.). Example: 300 transactions

per second.

There can be several levels of load:

o Peak number per hour or per second is typically used for system

sizing.

o Normal load levels are used to plan on-going support requirements.

o Low load levels are used to plan the percentage of resources which

can be taken off-line for repairs and upgrades.

Measures of Variation

The average measures the central tendency in a group of numbers. For

example, the average of 10, 20, and 30 is 20. This is calculated by adding all

41

Load Runner

three numbers together (10 + 20 + 30 = 60) and dividing this sum by the

number of items added together (in this example, three).

Standard Deviation

Standard deviation is a statistical measure of the extent of variation in a set of

data. The standard deviation tells you the extent of the spread of individual

items around the average.

The number in the 90% column is shown after a run is completed in the

Analysis program, not in the Controller) because it is calculated by sorting all

items by value obtained during the run, then identifying the value

associated the item at the top 10% of all items.

References

1. http://www.aptest.com/resources.html

2. http://www.sql-server-performance.com

3. http://www.mercuryinteractive.com/

4. http://www.qaforums.com

42