empirical software quality engineering: experience and recent development jeff tian, phd, pe...

Empirical Software Quality Engineering:Experience and Recent Development

Jeff Tian, PhD, PE Associate Director, NSF Net-Centric I/UCRCProfessor, CSE & EMIS, SMU, Dallas, Texas, USA

西北工业大学计算机学院陕西省百人计划特聘教授

Presentation given on 2013-07 at Tianjin University/天津大学

Copyright © 2010 NSF Net-Centric I/UCRC. All rights reserved.

2

Agenda

Quality and Empirical Observations

Quality Improvement Risk identification and prioritization

Risk resolution and quality improvement

Empirical Validation

New Trends and Directions Net-Centric, Service-Oriented, Cloud Computing, etc.

Questions, comments…

Net-Centric Software & Systems I/UCRC2013/07@Tianjin University, China

Empirical Software Quality EngineeringJeff Tian, SMU & 西工大

3

Software Quality

• Most intuitively associate with low defect• Different perspectives:

• user, service provider, infrastructure/data center, developer, tester, manager, 3rd party, etc.

• Different attributes: • capability, reliability, usability, safety, etc.

• Could be domain specific, e.g., • reliability, security, and usability for the web domain

• Grouping of quality attributions possible, e.g., • Dependability include reliability, availability, security,

etc.• This presentation focuses on reliability and

related quality attributes



4

Quality Attributes/Distribution for an IBM Product


Net-Centric Software & Systems I/UCRC

• ODC (orthogonal defect classification) categories

• Observation: highly uneven distributions

2013/07@Tianjin University, China

5

Quality Attributes/Distribution for an IBM Product



• Dependability attributes (rows)

• Stakeholders of information systems (colomns)

• Observation: highly uneven (qualitative) interests/importance


6

Quantification of Software Quality

Some quality attributes may be quantifiable, e.g.,

• Reliability: probability of failure-free operations for a specific time period or input set under a specific environment• Quantified: probability of ….• Customer perspective: failure view• Time, input, and environment characterization

• Others may not be quantifiable, e.g.,• Safety may be defined as the “property of being accident

free” (Leveson, “Safeware”, Addison-Wesley, 1995)




7


• Quantification requires data and measurement• Defect measurement• Other measurement

• Defect concepts and measurement• Failure: behavioral deviation from expectation• Fault: internal problem in software that may lead to

failure• Error: missing/incorrect human action leading to fault

injection• Collectively referred to as defect

• Example of reliability measurement:• Failure measurement• Time or input space measurement




8

Quantification: Measurement



• Fault distribution by type (fault class) for a large telecom system

• Histogram or “Pareto chart”

• Pareto principle or 80:20 rule originator

• Observation:• Highly skewed

distribution


9




• Defect & complexity measurement for an IBM product

• Box plot used• Mean• Distribution

boundaries• Observation:

• Highly skewed distribution, again


10




• Cross referencing for SMU Engineering School web site

• Scatter plot used• # = dot size• Distribution

visualization• Observation:

• Highly skewed distribution, again


11




• Relating metric to change for Mozilla

• Scatter plot to visualize distribution and correlation

• Observation:• skewed

/clustered distribution, again


12




• Number of transactions processed for each test run for an IBM RDBMS product

• Plot used• Time/sequence• Distribution

• Observation:• Highly skewed

distribution, again


13


• Some quality attributes may not be quantifiable, e.g.,• Safety may be defined as the “property of being accident

free” (Leveson, “Safeware”, Addison-Wesley, 1995)

• But, some quantitative characterization still possible:• Empirical data: about 90% of software safety problems can be traced to

interface/interaction related problem sources

• Hazard analysis to allow selective hazard resolution• Risk analysis: prob(hazard), prob(hazard-accident link), worst-damage

• Also, importance rating of individual dependability attributes by stakeholders example earlier




14

Quantifiable Quality Improvement

• Need a baseline• Defect measurement: distribution and characterization• Other measurement: internal char. + external relations

• Improvement paradigms• QIP by Basili: baseline-change assessment-implement• Process maturity work from SEI: moving up CMM/CMMI

levels• Other approaches

• Key observation based on empirical evidence: • non-uniform distribution, non-uniform risk• Implication: risk-based approach should be more

effective and efficient than other approaches, if….




15

What Is Risk?

• Most intuitively associate with undesirable, unacceptable, or less desirable consequences• Low reliability• High defect (or defect density)• Possible safety problems (and their likelihood)• Others

• Two definitions: mean vs. variance• Mean used mostly in daily life, and also commonly

used in computing and software engineering• Variance to capture (problems associated with)

uncertainty in financial/insurance industries, etc.




16

What to Do about Risk?

• 80:20 rule or Pareto principle:• Highly uneven distribution, a key characterization

based on empirical evidence• see (representative) examples earlier

• Both consequences (risk itself) and contributing/associated factors/entities (risk factors)

• What to do: Risk identification and resolution• Associating risk with other entities

• e.g., relating defect density to complexity • What to do after identification?

• Remedial/preventive actions




17

Risk Types/Perspectives

• Similar to quality perspectives, primarily internal vs. external risks

• External risk to customers/users: reliability risks, etc.• Reliability affected by

• Internal faults: count, density, distribution• Usage, characterized by OP (operational profiles)

• Risk-based reliability improvement• Internal risk to software organizations:

• Defect risk identification & reduction.• Metrics-defect predictive modeling.• ODC and extensions.




18

Agenda









19

Empirical/Risk-Based Quality Improvement

• Understanding the application domain and data

• Selecting quality/risk perspectives and related attributes (risk factors)

• Quantification and measurement

• Risk identification

• Risk resolution

• Empirical validation

• Implementation and deployment



20

Related Approach: Basili’s QIP/GQM/EF

• QIP (quality improvement paradigm):• Step 1: understand baseline• Step 2: change then assess impact• Step 3: package for improvement

• EF (experience factory), separate from development organization, to support QIP implementation

• GQM (goals/questions/metrics) to guide measurement/analysis activities

• Our ESQE: more streamlined process with more attention to empirical data/analysis/validation and focus on risks and closing the loop



21

Our ESQE: Evolution



Our ESQE:

•streamlined process

•empirical data/analysis/validation

•risk focus

•close loop

22

Our ESQE: Details



23

Approach: Risk Identification/Resolution

• Risk identification (where to focus)• Associating risk with other entities

• Risk = what we care about, trying to avoid• Others might be readily available or available early• Risk areas and their characteristics

• Establishing predictive relations• Many techniques can be used (later, with examples)

• Risk resolution (what to do)• Remedial actions for current project• Preventive/proactive actions for future projects

• Focus on high-risk/high-leverage areas




24

Risk Identification Techniques

• Qualitative risk identification techniques • Causal analysis: human intensive root cause analysis and variations• Risk indices, etc.• Suffering from various shortcomings, and not the focus of this

presentation

• Quantitative risk identification techniques:• Old/traditional statistical: correlation, regression, etc.• New (statistical): PCA/DA, TBM, etc.• AI/learning: NN, OSR, etc.• Survey in Tian/SQE book• Focus of this presentation, with many examples to come




25

Application Domain• Government: defense, aerospace and aeronautics

• NASA/SEL (Software Engineering Laboratory)• Early work (pre-1992) at U. Maryland

• Axiomatic framework for complexity

• Selection procedure

• Used for better effort prediction, improved results from earlier work by Selby and Porter

• Recent work (more details later)• Lockheed-Martin: quality and productivity improvement

• Raytheon and Boeing: components and services improvement



26

Application Domain• Commercial software

• IBM (1992-1995): in house work • IBM-SMU collaboration: continuation after 1995• Other companies: HP, TI, etc., associated with our NSF Net-

Centric I/UCRC

• Application of risk-based approach:• UBST (usage-based statistical testing) guided by OP

(operational profiles) for compilers• Reliability modeling and improvement for DBMS etc.• ODC (orthogonal defect classification) and extensions• Defect/metrics analysis and modeling



27

Application Domain• Telecommunications software and systems

• Nortel, Verizon, and telecommunications companies affiliated with NSF Net-Centric I/UCRC

• Application of risk-based approach to Nortel (pre-2002):• Some replication of IBM work

• formal hypothesis testing HC-HD?• characterization of HD modules

• Application to Verizon:

• E-commerce service quality improvement



28

Application Domain• Web-based systems

• Web quality characterization and measurement• Web testing, particularly usage-based, risk-focused• Diversity: .edu, .org, .com, social networking, e-commerce, etc.

• Open source software• Some replication of IBM/Nortel work

• Also in connection to web-based systems• Safety-critical systems:

• Nuclear industry case studies• Also in connection with government/DoD/NASA work

• Others （ new trend/direction later)



29

Risk/Reliability Focus: Important Usage

• Focus on functions/modules/features with• High usage frequency and importance

• Non-uniform testing effort

• Usage-based statistical testing (UBST)

• Other focused quality assurance, etc.

• UBST = OP-guided testing• Capture user/usage information• Usage model = Operational prole (OP)• SRMs: Testing results to estimate reliability• New applications in web, etc.



30

Risk/Reliability Focus: Important Usage



Traditional testing techniques: – functional (black-box) or structural (white-box) coverage

New testing techniques: – usage-based statistical testing (UBST)

– risk-based testing, etc.

Testing effectiveness/efficiency captured by defect discovery profile

desirable: curve bent towards upper-left corner

Claim: Risk-based testing more effective/efficient


31

UBST with Operational Profiles (OP)



• Web usage modeling• High-level: Musa’s

operational profile • Item and prob.• Sorted• Focused

testing• Medium-level• Low-level

• Example OP for SMU/SEAS


32

UBST Guided by OP



• Web usage modeling• High-level• Medium-level:

Markov chains (UMMs)

• Navigation• Transition

probabilities• Low-level

• Example OP for SMU/SEAS


33

UBST for Web Applications• Top-level: Musa’s operational profile (list of operations/hits sorted by hit count)

• Middle-level: Unified Markov Models (UMMs) to capture navigation

information for major operations and/or clusters of web-based functions

• Bottom-level: Pre-existing coverage testing of individual web components




34

Risk-Based Testing: TBRM => Failure Clusters

Tree-based reliability model(TBRM, Tian 1995)



– Relate input and timing to failures by

recursive

partitioning

– Subset reliability id. low reliability subsets for remedial actions

=> reliability

improvement


35

Risk-Based Testing: TBRM for IBM Products



Commercial software products from IBM– A, B, C: TBRM not used

– D: TBRM in system test

– similar pattern for

subsequent products Normalized comparison:

– Compare shape only

– Quantitative ρ:

purification level

D: ρ=0.94

A,B,C: ρ=0.65, 0.53, 0.45


36

Risk-Based Testing of Web Applications

Web sites and web-based applications become important part of modern life and business– Massive user population

– Initially document-focused

– More “computation” capabilities added in e-commerce/cloud-computing and other new web-based applications

– Reliability, security, and usability as primary quality attributes

– Heterogeneous components: diverse types of testing Risk-based web testing:

– Web log and defect data analysis

– Testing prioritized by defect rate ranking



37

Risk-Based Web Testing Techniques Data sources

– Web logs: Server access and error logs, application etc.

– Defect database and repositories

– Additional data collection possible: e.g., ODC (orthogonal defect classification) adapted to the web (Ma and Tian, 2007)

Risk analysis and identification: – Risk score = defect rate, two variations

#failures/#accesses #unique-failures / #files/documents (or #unique-accesses)

– Testing prioritized by defect rate ranking

– Combine data from multiple sources (Li/Alaeddine/Tian, 2010)



38

Test Prioritization Using Combined Data

• Fault view: unique failures from web logs

• Failure view: impact/usage model => falures from faults



39

Risk/Safety Focus: Problem Sources



• Physical and logical frames

• Focus on interface and interaction problems

• Empirical evidence for it


40

Risk/Safety Improvement



• Analyze sources of hazard: frame inconsistencies and sub-types

• Derive systematic assertions or prescriptive specifications

• Dynamically check these assertions


41

Risk Focus: Defect-Prone Areas



• Analyze metrics-defect relations• Correlation analysis• Regression models:

linear model here• Impact: behavior

modification (later: HT to confirm)

• Shortcomings: many, leading to other risk id. models


42

Risk Focus: Defect Reduction



• TBDM (Tree-based defect model)

• High-defect clusters

• Characterization• Lead to

remedial actions

• Drive quality improvement


43

Agenda









44

General Validation Issues• Case studies: empirical basis

• In lab: controlled environment

• In production/operational environment

• Controlled (scientific!) experiment/simulation• Experimentation: role of research labs

• Simulation: less expensive, some web results presented earlier

• Data and analysis via analysis of variance, hypothesis testing, etc.

• Implementation and deployment

• All results need to be carefully analyzed



45

Validation: Result Analysis• General analysis

• Distribution comparison: mean, variance, etc.• Trend analysis, etc.• Careful with causal relation vs. accidental correlation

• Formal hypothesis testing (HT)• More rigorous, powerful and better statistical validity

• HT study by Koru and Tian (TSE 2005):• High-defect (HD) modules vs. high-complexity (HC) modules• HD and HC statistically different• Complexity ranking of HD: 60-80%



46

Validation: HT by Koru/Tian (TSE 2005)



|z| critical values at 1.96 and 2.58 for significance levels of 5% and 1% respectively=>All null hypothesis rejected

47




|z| critical values at 1.96 and 2.58 for significance levels of 5% and 1% respectively=>All null hypothesis rejected

48




HC vs HM clusters=>Majority HC different from HM but inconclusive HT result;HC metrics ranking around 80th percentile

49

Risk-Based Testing: Validation



Commercial software products from IBM– A, B, C: TBRM not used

– D: TBRM in system test

– similar pattern for

subsequent products Normalized comparison:

– Compare shape only

– Quantitative ρ:

purification level

D: ρ=0.94

A,B,C: ρ=0.65, 0.53, 0.45


50

Risk-Based Web Testing: Experiments Diverse web sites/applications

– Academic web site: seas.smu.edu– Open source project site: kde.org– Small company catalog showroom web site– Large telecommunications company service ordering site– Social networking web site(s)

Experimental setup: training (1st ½) vs. testing (2nd ½) data– Training data log/defect analysis: test prioritization– Testing data: simulated behavior by processing actual data

Defect profile compared to those based on coverage/checklists/hierarchies/etc.



51

Risk-Based Web Testing: Validation



Academic web site:

seas.smu.edu– risk-based testing

better than

directory level:increasingdecreasingrandom

dictionary order + reverse order


52




Open source web site kde.org– risk-based testing better

– additional analysis pre/post

significant change


53




Small company catalog showroom on the web– risk-based testing better than other orders (similar results)


54




Large telecommunications company online service ordering– risk-based testing better (similar results)


55




Risk identification for a social networking web site

Risk AreaError number(Training)

Error Percent(Training)

Error number(Testing)

Error Percent(Testing)

None 0 0% 11 10.48%Crash 5 1.53% 3 2.85%Function Failure 266 81.35% 74 70.48%Function Workaround 36 11.00% 5 4.76%User Interface 20 6.12% 8 7.62%minorreq 0 0.00% 4 3.81%All 327 100.00% 105 100.00%

Social networking web site– Risk-based testing better than other orders in later ranking too


56

Agenda









57

New Trend and New Research• Net-centric operations (NCO) and net-enabled

capabilities (NEC)• US DoD and defense industry (UK and Europe too)• Goal is to achieve information superiority through (dynamic)

integration of people, (changeable, often hostile) environment, and (fluid) infrastructure

• Service-oriented computing and service-oriented architecture (SOA)• Commercial industry driven, particularly web-based business• Similar concerns as NCO, quality/performance/etc. through SLA

(service level agreement)



58

New Trend and New Research• Cloud computing and related software/system issues:

• Service provided via cloud instead of locally

• Shares many similar concerns with NCO and particularly SOA, but with more of a focus over the overall infrastructure instead of services only

• Recent (starting Sept. 2011) project funded by NSF• Instrumentation to measure dependability and quality of cloud

computing systems

• Collaboration among SMU, UTD and UNT, founding members of NSF Net-Centric I/UCRC

• Total funding $1.4+ million over 3 years



59

New Trend and New Research• Many common issues among NCO, SOA, and CLOUD:

• Complex, dynamically composed systems

• Integration of diverse components (including people too)

• Quality/dependability measurement, evaluation, and assurance at both the component and system level

• Representing concerns of our NSF I/UCRC member companies:• NCO/SOA/CLOUD for Lockheed-Martin, Raytheon, Boeing, etc.

• SOA/CLOUD for EDS, TI, Cisco, etc.



60

Existing Solutions• V-model for products to reduced V-model for

components used in NCO/SOA/CLOUD



61

Existing Solutions• Support for measurement, analysis, and reuse

through experience factory for whole products




62

Our Solution for NCO/SOA/Cloud

• Evaluator for NCO/SOA/CLOUD system components and services based on multiple perspectives and realistic application scenarios

• A testbed for application scenario modeling/simulation• Normal scenario captured in operational profiles for testing• Perturbation and extreme/hostile condition modeling via boundary

extension, mutation, fault inject, security threat simulation, etc.

• A multi-facet/perspective evaluator• External quality/dependability attribute identification/evaluation• Internal quality characterization and empirical based mapping• Multivariate optimization using data envelopment analysis• Quantify relative value and tradeoffs for optimal decision making




63

Our Solution: Tool Suite Architecture




64

Our Solution: Experimental Plan

• Overall: incremental/iterative approach• Risk-/opportunity-based identification of increments/iterations

• Continuous adjustment and improvement

• Each increment/iteration includes the following steps• Stakeholder identification

• Stakeholder concerns and priorities to derive context-sensitive external quality/dependability attribute definitions as well as value assessments

• Data collection and testbed/evaluator construction

• Experimental self-validation




65

Quality/Dependability Evaluation: Reliability

• Operational reliability captured by failure rate per time-unit or op. instances

• Reliability growth potential evaluated by unique failure sequence analysis

• Time/operation measurement: #users, #sessions, #hits, #bytes for web services




66

Defect Analysis and Prioritization• Defect analysis/prioritization based on web logs and other project documents

• Adaptation of ODC (orthogonal defect classification) to SOA/NCO/etc.

• Key attributes: response code, file type, trigger/referrer, time,

• Work by Alaeddine and Tian


0%5%

10%15%20%25%30%35%

% o

f T

ota

l

Fault class



67

Multivariate OptimizationDEA (data envelopment analysis) in HASE’2011 paper by Siok &Tian


InputsInputs OutputsOutputs

Software Reliability At ReleaseDefect Density-1

Defects Removed

EfficiencyEfficiencyOutput / Input

Software Total SizeSoftware Change SizeSoftware Development HoursSoftware Test HoursSoftware Schedule

InputsInputs OutputsOutputs

Software Reliability At ReleaseDefect Density-1

Defects Removed

EfficiencyEfficiencyOutput / Input

Software Total SizeSoftware Change SizeSoftware Development HoursSoftware Test HoursSoftware Schedule



68

Summary









This approach is proven to be effective

This new research looks promising

69

Agenda









empirical software quality engineering: experience and recent development jeff tian, phd, pe...

Documents