issre2010 malik
TRANSCRIPT
1
Pinpointing the Subsystems Responsible for Performance Deviations In a Load Test
Haroon Malik, Bram Adams & Ahmed E. HassanSoftware Analysis and Intelligence Lab (SAIL)
Queen’s University, Kingston, Canada
2
Large scale systems need to satisfy performance constraints
3
TIME SPENT…..
ANALYSTS SPENT CONSIDERABLE TIME DEALING WITH PERFORMANCE BUGS
4
LOAD TESTING
5
Environment Setup Load test execution Load test analysis Report generation
CURRENT PRACTICE1 2 3 4
6
Environment Setup Load test execution Load test analysis Report generation
CURRENT PRACTICE
1 2 3 4
7
2. LOAD TEST EXECUTIONMONITORING TOOL
LOAD GENERATOR- 1 SYSTEM PERFORMANCE REPOSITORY
LOAD GENERATOR- 2
8
Environment Setup Load test execution Load test analysis Report generation
CURRENT PRACTICE
1 2 3 4
9
Environment Setup Load test execution Load test analysis Report generation
CURRENT PRACTICE
1 2 3 4
10
LOAD TEST PASSFAILx
3. LOAD TEST ANALYSIS
MANUAL
12
LARGE NUMBER OF PERFORMANCE COUNTERS
13
LIMITED TIME
14
LIMITED KNOWLEDGE
15
WE CAN HELP ANALYSTS:
Decide if a performance test passed or failed CSMR 2010
Identify the subsystems which violated the performance COMPSAC 2010
Pinpoint the subsystems that are likely the cause of performance violation ISSRE 2010
16
Automated
Methodology to PINPOINT
Likely Cause of performance deviations
17
METHODOLOGY STEPS1 2 3 4
Data Preparation Crafting Performance Signatures Identifying Deviations Pinpointing
18
PERFORMANCE COUNTERS ARE HIGHLY CORRELAED
CPU DISK (IOPS)
NETWORK
MEMORY
TRANSACTONS/SEC
19
Principal Component Analysis (PCA) Explains most of the counter data with minimal
information loss Removes the noise in the counter data
Influential CountersCounter Elimination: Norman cut-off criteria Counter Ranking
2. CRAFTING PERFORMANCE SIGNATURES
Ide
Commits/Sec
Writes/Sec
CPU Utilization
Database Cache % Hit
Subsystems Base-Line Load Test - 1 DeviationMatch
0.41
0
0.01
PINPOINTING
Avg. Pair wise correlation ()SUB- A SUB-B
SUB-C SUB-D
0.8
0.7
0.90.70.50.6
22
4. PINPOINTINGSUB- A SUB-B
SUB-C SUB-D
0.8
0.7
0.90.70.50.6
Avg Pair-wise correlation ()
SUB LoadA 0.75B 0.77C 0.80D 0.77
23
4. PINPOINTINGSUB- A SUB-B
SUB-C SUB-D
0.8
0.7
0.90.70.50.6
Avg Pair-wise correlation ()
SUB Load Baseline Dev %A 0.75 0.87 0.12 13.0B 0.77 0.82 0.05 6.01C 0.80 0.94 0.14 14.8D 0.77 0.88 0.11 12.5
24
4. PINPOINTINGSUB- A SUB-B
SUB-C SUB-D
0.8
0.7
0.90.70.50.6
Avg Pair-wise correlation () Pinpointed
SUB Load Baseline Dev %C 0.80 0.94 0.14 14.8A 0.75 0.87 0.12 13.0D 0.77 0.88 0.11 12.5B 0.77 0.82 0.05 6.01
25
CASE STUDY
26
DELL DVD STOREComponents of Test Environment
Load GeneratorDatabase
ServerWeb Server (B)
Web Server (C)
Web Server (A)
Load
Gen
erat
ors
Performance Logs
Performance Monitoring Tool
27
Base 4-X Dev %Web-1 0.87 0.72 0.15 17.1Web-2 0.88 0.82 0.05 6.46Web-3 0.89 0.83 0.06 7.03DB 0.78 0.73 0.05 6.92
EXPERIMENT-14X- LOAD ON WEB-1
Base CPU Dev %Web-1 0.87 0.69 0.18 21.1Web-2 0.88 0.80 0.08 9.29Web-3 0.89 0.80 0.09 10.6
DB 0.78 0.73 0.05 7.24
Base CPU Dev %Web-1 0.87 0.83 0.03 4.28Web-2 0.88 0.83 0.04 5.04Web-3 0.89 0.84 0.05 6.14DB 0.78 0.78 0.08 10.4
Base MEM Dev %Web-1 0.87 0.81 0.06 7.42Web-2 0.88 0.75 0.13 14.9Web-3 0.89 0.81 0.08 9.49DB 0.78 0.7 0.087 11.0
EXPERIMENT-2CPU STRESS ON WEB-1
EXPERIMENT-3CPU STRESS ON DB
EXPERIMENT-4MEMORY STRESS ON WEB-2
28
1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 1610.4
0.5
0.6
0.7
0.8
0.9
1.0 Test- A (Base-line) Test-B Test-C Test-D Test-E
Performance Counters
Coun
ter I
mpo
rtan
ce
ENTERPRISE APPLICATION
29
WHY ARE TEST-D & E DIFFERENT?
30
WHY ARE TEST-D & E DIFFERENT? PINPOINTED – Among 6 subsystems for each test,
web server is likely the cause of performance deviation in both tests
Signature counters of web server notably deviated from the baseline:The packet Outbound Discarded,Packet Sent/SecMessage Queue Length
Network Problem – No connectivity to the databaseWeb server Under Stress
31
LIMITATIONS
Our methodology can only point *A* subsystem that is likely the cause of performance deviation for a load test.
Our methodology can not be generalized to other domains such as network traffic and security monitoring.
32