Jan. 14, 2013

Gum-Ho Choe,

Korea Laboratory Accreditation Scheme (KOLAS)

Korean Agency for Technology & Standards (KATS)

Ministry of Knowledge and Economy(MKE)

Hong Kong ITC/HKAS Seminar

·

I. Nationally Recognized SW Program

II. Results of Survey on APLAC’s ABs

III. Accreditation Program in ABs

IV. Future Work of SW Testing

• Two Track System of Software Testing in Korea

• 1st ; Nationally Recognized Testing & Certification

• 2nd ; KOLAS Accredited Testing Laboratories

Process

Certification

ISO/IEC 15504(SPICE) CMMi SP(Software Process)

System

Certification

ISO 9001

TL9000

Product

Certification

SW Product Certification - GS Certification - Conformance Certification : RFID, Home Network etc.

Nationally Recognized SW Testing & Certification

SW product (include process doc.)

- CC Certification - SIL Certification (IEC61508) - V&V etc

ISO/IEC JTC 1 SC7 Standards Collection

1. System Certification

ISO 9001 certification

- to evaluate the provider’s quality system and admit quality guaranty capacity - to present 20 requirements for quality guaranty and test establishing quality system available to each customer.

System Certification

TL 9000 certification - TL 9000 is a quality management practice designed by the QuEST Forum. - It was created to focus on supply chain directives for the international telecommunications industry

- CB: NIPA(National IT Industry Promotion Agency)

- Process improvement approach that provides organizations with the essential elements of effective processes that ultimately improve their performance. - Can be used to guide process improvement across a project, a division, or an entire organization - SP is domestic certification in Korea based on ISO/IEC 15504. : for middle and small companies in Korea * IT Companies should be certificated to bid to public institutes.

: SP, CMMi, SPICE

Process Certification

SP Mark

2. Process Certification

Level 1

Level 2

Level 3 Process measured

and control & process

improvement

Process characterized for

project & organization

Processes poorly

controlled and reactive

Process improve

project level

Process improve

organization level

Maturity Levels(SP)

Product

Certification

GS(Good Software) Certification

CC(Common Criteria) Certification

SIL(Safety Integrity Level) Certification

V&V(Validation & Verification)

- Medical Area, Nuclear Area etc

Product Certification

Product Evaluation - except product development process - evaluate only the product itself

- GS certification

Evaluation of product including process documents

- overall lifecycle evaluation such as requirements specification, code and Execution file, so on. • Security certification : CC(Common Criteria) • Safety certification : SIL(Safety Integrity Level)

ISO/IEC 29119 IEEE 829 ISTQB Syllabus …

TMMi/TPI

Requirements Analysis

System Design

Architecture Design

Module Design

Coding

Unit testing

Integration Testing

System testing

Acceptance Testing

Software Testing and Related Standards

Measurement of Internal Quality

ISO/IEC 9126-1, ISO/IEC 25022

Measurement of External Quality

ISO/IEC 9126-2, ISO/IEC 25023

Measurement of Quality in Use

ISO/IEC 9126-3, ISO/IEC 25024

ISO/IEC 14598 series

Requirements Analysis

System Design

Architecture Design

Module Design

Coding

Unit testing

Integration Testing

System testing

Acceptance Testing

Software Measurement and Related Standards

SW Testing vs. Measurements

S/W Test

finding defects

providing confidence in the

product

providing insight in quality and

risks

Measurement

Identifying the Risks to System

providing quality in the product

using the test result

SW Measurements Methods on GS Certification

Informal

Risk Analysis

ISO 9126/25000 series

Quality Analysis

Failure Mode and Effect Analysis

Start with the classic quality risk categories

Start with six main quality characteristics

Start with categories, characteristics, or subsystems

Functionality, states and transactions, capacity and volume, data quality, error handling and recovery, performance, standards and localization, usability, etc

Functionality, Reliability, Usability, Efficiency, Maintainability, Portability(FRUEMP), then decompose into key subcharacteristics for your system

Key stakeholders list possible failure modes, predict their effects on system, user, society, etc., assign severity, priority, and likelihood, then calculate risk priority number(RPN)

Set priority for testing each quality risk with key stakeholders

Set priority for testing each subcharacteristic with key stakeholders

Stakeholders use RPN to guide appropriate depth and breadth for testing

GS Certification

기능성 신뢰성 사용성 효율성 유지보수성 이식성

적합성 정확성 상호운용성

보안성 준수성

시간반응성 자원효율성

준수성

Overall Quality Model

12 17 25 22 15 10

주특성

부특성

Requirement Basis Related Doc. note

System

Requirements

RFP 2-7D0-J222-001

56 M/W Technology

Selection Criteria

Technical

Documents

(Company)

SW Function User Manual 138

Quality

Requirements

ISO/IEC 9126-2, 25051

Evaluation Module for Enterprise S/W

62

Total Requirements are 286.

Example- Inchon Airport Information System

16

Focusing on Industries of SW Testing

Information

Technology

Convergence

Automobile

Aerospace

Telecommunication

Construction

Healthcare Defense

Shipbuilding

Machinery

Testing & Certification Area of IT Convergence

Conformance IEC 61850 etc

Sector Standards

Reliability

IEC 60068 etc

Safety ISO/IEC 26262 IEC 61508 etc

Interoperability Sector Standards etc

Security ISO/IEC 15408

Sector Standards etc

Quality/Performance IEC62278, ISO/IEC25000

Sector Standards etc

Product/ System

ABs ACLA

SS A2LA BoA HKAS IANZ IAJAPAN KOLAS NATA SAC SCC SM CNAS

1. Testing

Labs o(2) o(11) X X X o(6) o(9) o(15) X X o o(80)

2. Guides o o o o o o o

3. SR ? NA NN o o o o o

4. Experts,

etc NN NA o NA o NA o o o

1. Accreditation service,

2. ISO/IEC 17025 + Supplement requirement(SR) are enough

3. Necessity of comprehensive SR

4. Experts; Workshop, Training, etc

o : Yes, X : No, ? : Hard to answer, NA : No Answer, NN : Do Not Need, * NVLAP (34), NABL(1)

* SM wants to take training course by leading ABs such as A2LA, NVLAP, NATA, etc., and hold

workshop

. Circulated through 34 ABs : 12 ABs answered ( 35 % : answering rate)

Current Status

KOLAS has accredited 9 testing laboratories for SW

Accredited Laboratories

No. Lab. Name Accreditation

Date

Accreditation Scope

(Sub Major Discipline)

KT 005 Korea Testing

Certification

June 10, 2008 Embedded SW for Smartcard

KT 009 Korea Testing

Laboratory

September 30,

2010

SQuaRE, COTS, Common Criteria

for IT Security Evaluation

KT 519 Financial Security

Agency

March 19, 2012 ISO/IEC 25051

ISO/IEC 9126-2

KT 519

Accredited Laboratories

No. Lab. Name Accreditation Date Accreditation Scope

(Sub Major Discipline)

KT 167 Telecommunications

Technology

Association

May 14, 2009 Common Criteria for IT Security

Evaluation

KT 327 Korea System

Assurance

June 27, 2007 Common Criteria for IT Security

Evaluation

KT 402 Korea Security

Evaluation Laboratory

Co. Ltd.

April 24, 2009 Common Criteria for IT Security

Evaluation

KT 448 ICT Korea Ltd. April 26, 2010 ID Card; ISO/IEC 7816-3, 10373-3,

ISO/IEC14443-2~-4

KT 463 KISA November 16, 2010 ISO/IEC 24709-1(IT-Conformance

testing for the biometric application

programming interface

KT 122 SGS Korea Aug. 23, 2012 IEC 61508-3 (Functional Safety)

ISO 26262-6 (Safety Management

System)

KT 122 ; SGS KOREA

KT 167 ; TTA

KT 463 ;KISA

25

• CC Certification

• CC Consulting

• Smart Grid and

Industrial Network

Security Test

Security Evaluation

Team

• GS Certification

• SW KOLAS(ISO/IEC

17025)

Testing Service

• Process Certification

(SPICE, CMMi)

SW Evaluation

Team

IT System Evaluation

Team

• IEC61508 (Safety) SIL

certification

• Mission-Critical SW

performance and

Reliability Test

• SW V&V etc

Major Work of KTL (Korea Testing Laboratory) Recognized & Accredited

by MKE & KOLAS, Respectively

KT 009

Information Technology Security Testing

Common Criteria Testing ; 7 labs

Cryptographic and Security Testing ; 21 labs (foreign 11 EA)

Healthcare Information Technology Testing ; 5 labs

U.S. A. NIST NVLAP

U.S. A. A2LA

* 11 labs accredited

Information and Communications Technology

NATA's Information and Communications Technology field

provides accreditation for a diverse range of software and

hardware testing in such areas as gaming systems,

information security evaluations and healthcare software.

* 15 labs accredited

1. Focusing on Product Quality and Security

2. Extending to Safety Management System &

Healthcare Information Technology

Extending to Safety Management System (SMS)

Based on ISO 26262, Technical Regulations, etc.

Current Status and Future Work

Based on CC, ISO/IEC 9126, 25000, and IEC 61508, etc

Embedded SW in fields of IT convergence

Sep. 12, 2011

Gum-Ho Choe, Convenor

Working Group on Accreditation of Software Testing,

APLAC Technical Committee

Reference 1

Dec. 3, 2012

Gum-Ho Choe, Convenor

Working Group on Accreditation of Software Testing,

APLAC Technical Committee

Reference 2

Thank You! Gum-Ho CHOE gumho.choe@gmail.com,