메트릭스 솔루션의 새 양식

Software Testing Best Practices

2007년

7월

2일

Copyright© 2007 Metrics Solutions. Work Smarter, Not Harder. www.metricsolution.co.kr 2

목 차

1.

Groups of Best Practices ………………………………………………………………………………………………

3

2.

The Basic Practices

………………………………………………………………………………………………………

4

3.

Foundational Practices

…………………………………………………………………………………………………

6

4.

Incremental Practices

……………………………………………………………………………………………………

8

5.

연락처

……………………………………………………………………………………………………………………………

11


Groups of Best Practices

The Basic Practices

Foundational Practices

Incremental Practices

1.

Functional Specifications

2.

Reviews and Inspections

3.

Formal Entry and Exit Criteria

4.

Functional Test –

Variations

5.

Multi-platform Testing

6.

Internal Betas

7.

Automated Test Execution

8.

‘Nightly’

Builds

1.

User Scenarios

2.

Usability Testing

3.

In-process ODC Feedback Loops

1.

Teaming Testers with Developers

2.

Code Coverage

3.

Automated Environment Generator

4.

Testing to Help Ship on Demand

5.

State Task Diagram

6.

Memory Resource Failure

Simulation

7.

Statistical Testing

8.

Semiformal Methods

9.

Check-in Tests for Code

10.

Minimizing Regression Test Cases

11.

Instrumented Versions for MTTF

12.

Bug Bounties

4.

Multi-release ODC/Butterfly Profiles

5.

Requirements for Test Planning

6.

Automated Test Generation


The Basic Practices

Functional Specifications

•

A key part of many development processes.

•

External view of an object or a procedure indicating the options

by which a service could be invoked.

•

Testers use functional specifications to write down test cases from a black box testing perspective.

•

Test generation activity could happen in parallel with the development of the code.

Reviews and Inspections

•

One of the most efficient methods of debugging code.

•

Software inspection can easily provide a ten times gain in the process of debugging software.

Formal Entry and Exit Criteria

•

Every process step, be it inspection, functional test, or software design, has a precise entry and

precise exit criteria.

•

The practices allow much more careful management of the software

development process.

Functional Test – Variations

•

A variation refers to a specific combination of input conditions

to yield a specific output condition.

•

Writing down functional tests involves writing different variations to cover as much of the state space

as one deems necessary for a program.


The Basic Practices

Multi-platform Testing

•

Many products today are designed to run on different platforms which creates the additional burden to

both design and test the product.

Internal Betas

•

The idea of a Beta is to release a product to a limited number of customers and get feedback to fix

problems before a larger shipment.

Automated Test Execution

•

The goal of automated test execution is that we minimize the amount of manual work involved in test

execution and gain higher coverage with a larger number of test cases.

‘Nightly’ Builds

•

The concept captures frequent builds from changes that are being

promoted into the change control system.

•

If a major regression occurs because of errors recently generated, they are captured quickly.

•

Regression tests can be run in the background.

•

The newer releases of software are available to developers and testers sooner.


Foundational

User Scenarios

•

As we integrate multiple software products and create end user applications that invoke one or a

multiplicity of products, the task of testing the end user features gets complicated.

•

It tests the product in the ways that most likely reflect customer usage, imitating what Software

Reliability Engineering has for long advocated under the concept

of Operational Profile.

•

One reduces the complexity of writing test cases by moving to testing scenarios than features of an

application.

•

This best practice should capture methods of recording user scenarios methods when specific failure

scenarios occurs.

Usability Testing

•

Usability becomes the final arbiter of quality.

•

Usability testing needs to not only assess how usable a product is but also provide feedback on methods

to improve the user experience and thereby gain a positive quality image.

In-process ODC Feedback Loops

•

Orthogonal Defect Classification (ODC) is a measurement method that uses the defect stream to provide

precise measurability into the product and the process.

•

One of the uses of ODC has been the ability to close feedback loops in a software development process.


Foundational

Multi-release ODC/Butterfly Profiles

•

A key feature of the ODC measurement is the ability at multiple releases of a product and develop a

profile of customer usage and its impact on warranty costs and overall development efficiencies.

•

The technology of multi-release ODC/Butterfly analysis allows a product manager to make strategic

development decisions so as to optimize development costs, time to market, and quality issues by

recognizing customer trends, usage patterns, and product performance.

Requirements for Test Planning

•

Requirements management and its translation to produce test plans is an important step.

Automated Test Generation

•

Almost 30% of the testing can be the writing of test cases. This

is a completely manual exercise and a

prime candidate for savings through automation.


Incremental

Teaming Testers with Developers

•

The close coupling of testers with developers improves both the test cases and the code that is developed.

Code Coverage

•

The concept of code coverage is based on a structural notion of the code.

•

Code coverage implies a numerical metric that measures the elements of the code that have been exercised

as a consequence of testing. There are a host of metrics: statements, branches, and data that are implied

by the term code coverage.

Automated Environment Generator

•

A fairly time-consuming task is the setting up a test environments to execute test cases.

•

Tools that can automatically set up environment, run the test cases, record the results, and then

automatically reconfigure to a new environment, have high value.

Testing to Help Ship on Demand

•

It changes the role of testing to one of providing excellent regression ability and working in late

changes that still do not break the product or the ship schedule.


Incremental

State Task Diagram

•

This practice captures the functional operation of an application or a module in the form of a state

transition diagram.

•

The advantages of doing so allows one to create test cases automatically or create coverage metrics that

are closer to the functional decomposition of application.

•

The difficulties have usually been in extracting the functional view of a product which may not exist in

any computable or documented form and producing the state transition diagram.

Memory Resource Failure Simulation

•

This practice addresses a particular software bug, namely the loss of memory because of poor heap

management or the lack of garbage collection.

Statistical Testing

•

The central idea is to use software testing as a means to assess

the reliability of software as opposed

to a debugging mechanism.

Semiformal Methods

•

A verification of a program as opposed to testing and debugging.

•

The verification methods are varied, some of which are theorem provers, while some of them simulation

against which assertions can be validated.


Incremental

Check-in Tests for Code

•

The idea of check-in test is to couple an automated test program (usually a regression test) with the

change control system.

•

This allows for an automated test run on recently changed code so that the chances of the code breaking

the build are minimized.

Minimizing Regression Test Cases

•

In organizations that have a legacy of development and of products that have matured over many releases,

it is not uncommon to find regression test buckets that are huge.

•

At the same time, it is often unclear as to which of these test cases are duplicative providing little

additional value.

•

One of the methods looks at the code coverage produced, and distill test cases to a minimal set.

Instrumented Versions for MTTF

•

If the product is instrumented so that failures are recorded and

returned to the vendor, they would yield

an excellent source to measure the mean time between failure of the software.

Bug Bounties

•

Bug bounties refers to our initiatives that charge the organization with a focus on detecting software

bugs. At times providing rewards too.

•

The experience states that such effort tend to identify a larger

than usual number of bugs.


연락처

감사합니다!

많은

문의를

부탁

드립니다

.

오 상헌

메트릭스

솔루션(Metrics Solutions)

서울시

강남구

대치4동

920-32번지, 202호

전화: 02-562-3749

FAX: 02-562-3749

휴대폰: 019-364-3749

E-Mail: [email protected]

Web: http://www.metricsolution.co.kr

mailto:[email protected]

메트릭스 솔루션의 새 양식

Documents