vsts 와 함께하는 소프트웨어 품질 보장 및 테스팅 기법 kim sung [email protected]...
TRANSCRIPT
VSTSVSTS 와 함께하는 소프트웨어 품질 보장와 함께하는 소프트웨어 품질 보장
및 테스팅 기법및 테스팅 기법
Kim SungKim Sung
[email protected]@gmail.com
http://blog.naver.com/skim21chttp://blog.naver.com/skim21c
Samsung ElectronicsSamsung Electronics
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 22
AgendaAgenda
SW QualitySW Quality
Software Development Life CycleSoftware Development Life Cycle
Test DesignTest Design
Test ProcessTest Process
Unit TestUnit Test
Load TestLoad Test
Visual Studio Team SystemVisual Studio Team System
Testing with VSTSTesting with VSTS
Testing Demo Testing Demo
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 33
SW Quality?SW Quality?
What’s the What’s the
SW QualitySW Quality??
ReliabilityReliability
FunctionalityFunctionality
SafetySafety
Ease of UseEase of Use
EconomyEconomy
SatisfactionSatisfaction
??????
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 44
SW Quality?SW Quality?
- 설치 후 기능이 정상적으로 동작하지 않는다설치 후 기능이 정상적으로 동작하지 않는다 ..
- 기능 추가를 하려고 했더니 많은 곳에서 에러가 난다기능 추가를 하려고 했더니 많은 곳에서 에러가 난다 ..
- 사용자 정보가 해킹 당했다사용자 정보가 해킹 당했다 ..
- 서버가 바이러스에 감염되었다서버가 바이러스에 감염되었다 ..
- 설치가 너무 어렵다설치가 너무 어렵다 ..
- 사용하기가 너무 어렵다사용하기가 너무 어렵다 ..
- 시스템을 사용하는데 너무 느리다시스템을 사용하는데 너무 느리다 ..
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 55
Factors for Customer SatisfactionFactors for Customer Satisfaction
Requirements are metRequirements are met
On Time DeliveryOn Time Delivery
With in BudgetWith in Budget
Technical SupportTechnical Support
High Quality ProductHigh Quality Product
CustomerCustomer
EngineerEngineer
InstallerInstaller
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 66
SW Development FactorsSW Development Factors
프로세스프로세스
기술기술 인적자원인적자원
ProductProduct SystemSystemServiceService
품질 향상품질 향상 생산성 향상생산성 향상
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 77
RequirementsRequirements
other, 50%
poor user input,13%
incompleterequirement,
12%
changingrequirement,
12%
poor technicalskills, 7%
poor staffing,6%
Requirements : 37%Requirements : 37%
성공적이지 못한 소프트웨어 프로젝트 요인성공적이지 못한 소프트웨어 프로젝트 요인
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 88
CodeCode
RequirementsRequirements
DesignDesignTest CaseTest Case
SW RequirementsSW Requirements
TraceabilityTraceability
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 99
RequirementsRequirements
결함 추가 및 발견 시기에 따른 평균 해결 비용결함 추가 및 발견 시기에 따른 평균 해결 비용
발견된 시기발견된 시기
추가된 시기추가된 시기 요구사항요구사항 아키텍쳐아키텍쳐 구현구현 테스트테스트 출시출시
요구사항요구사항 11 33 5-105-10 1010 10-10010-100
아키텍쳐아키텍쳐 -- 11 1010 1515 25-10025-100
구현구현 -- -- 11 1010 10-2510-25
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1010
RequirementsRequirements
“ “ 요구사항은 테스트 가능해야 한다요구사항은 테스트 가능해야 한다 .”.”
Requirement VerificationRequirement Verification
Test Case ATest Case A
Test Case BTest Case B
Requirement 1Requirement 1
Requirement 2Requirement 2
Requirement 3Requirement 3
Requirement 4Requirement 4
Requirement 5Requirement 5
TestedTested
UntestedUntested
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1111
ArchitectureArchitecture
Domain LayerDomain Layer
Infrastructure LayerInfrastructure Layer
Application LayerApplication Layer
ProtocolProtocolGeneratorGenerator
ProtocolProtocolParserParser
LoggerLoggerSenderSender
ReceiverReceiver CommCommManagerManager
ObjectObjectFactoryFactory DAODAO
ApplicationApplication
ConfigConfigManagerManager
GUI LayerGUI LayerGUIGUI
SecuritySecurity
ThreadThreadManagerManager
Component & Layer ArchitectureComponent & Layer Architecture
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1212
ArchitectureArchitecture
Interface DesignInterface Design
Domain Layer, Infrastructure LayerDomain Layer, Infrastructure Layer
ApplicationApplication
ISenderISender
IReceiverIReceiverIProtocolGeneratorIProtocolGenerator
IProtocolParserIProtocolParser
ILoggerILogger
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1313
ArchitectureArchitecture
Configurable ArchitectureConfigurable Architecture
Domain LayerDomain Layer
Infrastructure LayerInfrastructure Layer
Application LayerApplication Layer
ProtocolProtocolGeneratorGenerator
ProtocolProtocolParserParser
LoggerLoggerSenderSender
ReceiverReceiver CommCommManagerManager
ObjectObjectFactoryFactory DAODAO
ApplicationApplication
ConfigConfigManagerManager
GUI LayerGUI LayerGUIGUI
SecuritySecurity
ThreadThreadManagerManager
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1414
ImplementsImplements
programprogram faultfault
Code Inspection, Unit TestCode Inspection, Unit Test
errorerror
failurefailure
ExecutionExecution(testing)(testing)
debuggingdebugging
error, fault, failure?error, fault, failure?
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1515
Test DesignTest Design
Function1Function1
Function2Function2
Function1Function1
Function2Function2
Component 1Component 1 Component 2Component 2
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1616
Test DesignTest Design
Function1Function1
Function2Function2
ComponentComponent
DBDB
NetworkNetwork
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1717
Test DesignTest Design
Function1Function1
Function2Function2
ComponentComponent
DBDB
NetworkNetwork
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1818
Test DesignTest Design
ComponentComponent
Mock ObjectMock Object(Network)(Network)
FactoryFactory
FrameworkFramework
ConfigurationConfiguration
<Running Status = 0 or 1><Running Status = 0 or 1>
II
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 1919
Test DesignTest Design
Test Class1Test Class1
Test Class2Test Class2
Control Class1Control Class1
Control Class2Control Class2
ISenderISender
MockSenderMockSender SenderSender
FactoryFactory
Unit TestUnit Test
Unit TestUnit Test
Integration TestIntegration Test
NetworkNetwork
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2020
Test DesignTest Designpublic interface ISenderpublic interface ISender
{{
byte[] sendData(byte[] data);byte[] sendData(byte[] data);
}}
public class Sender : ISenderpublic class Sender : ISender
{{
public byte[] sendData(byte[] data)public byte[] sendData(byte[] data)
{{
byte[] receiveData = null;byte[] receiveData = null;
// Send Data// Send Data
// receive Data// receive Data
return receiveData;return receiveData;
}}
}}
public class MockSender : ISenderpublic class MockSender : ISender
{{
public byte[] sendData(byte[] data)public byte[] sendData(byte[] data)
{{
byte[] MockData = null;byte[] MockData = null;
// Make MockData// Make MockData
return MockData;return MockData;
}}
}}
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2121
Test DesignTest Designpublic class SenderFactorypublic class SenderFactory
{{
public ISender getInstance(int i)public ISender getInstance(int i)
{{
IISender net;Sender net;
if (i == 0)if (i == 0)
{{
net = new MockSender();net = new MockSender();
}}
elseelse
{{
net = new Sender();net = new Sender();
}}
return net;return net;
}}
}}
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2222
Test DesignTest Design
class ControlDeviceclass ControlDevice
{{
int controlDoor(byte[] key)int controlDoor(byte[] key)
{{
SenderFactory factory = new SenderFactory();SenderFactory factory = new SenderFactory();
ISender sender;ISender sender;
sender = factory.getInstance(config.Status);sender = factory.getInstance(config.Status);
byte[] result = sender.sendData(key);byte[] result = sender.sendData(key);
return 1;return 1;
}}
}}
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA
Refactoring Test DesignRefactoring Test Design
Legacy Legacy Code1Code1
RefactoringRefactoringCode1Code1
Legacy Legacy Code2Code2
RefactoringRefactoringCode2Code2
RefactoringRefactoring
Code3Code3
RefactoringRefactoring
Code4Code4
RefactoringRefactoring
Code5Code5
UIUI
Legacy SystemLegacy System
RefactoringRefactoring
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2424
ProcessProcess
Analysis & DesignAnalysis & Design
CodingCoding
TestingTesting
Test Test PlanPlan
Test Test CaseCase
Unit TestUnit Test IntegrationIntegrationTestTest
SystemSystemTestTest
AcceptanceAcceptanceTestTest
InstallationInstallationTestTest
PerformancePerformanceTestTest
StressStressTestTest
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2525
Test ProcessTest Process
Test DesignTest Design
SpecificationSpecificationOr SoftwareOr Software
SelectionSelectionCriteriaCriteria
Test Test ModelsModels
Test CasesTest CasesTest CaseTest CaseDesignDesign
ApplicationApplicationModelingModeling
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2626
Test ProcessTest Process
Test Execution and EvaluationTest Execution and Evaluation
TestTestOracleOracle
SoftwareSoftwareUnder TestUnder Test
Test Test ResultsResults
Test CasesTest Cases
TestTestExecutionExecution
TestTestEvaluationEvaluation
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2727
Test ProcessTest Process
Test PlanningTest Planning
Test Case GenerationTest Case Generation
Test environment developmentTest environment development
ExecutionExecution
Test results evaluationTest results evaluation
Problem reporting / Test logProblem reporting / Test log
Defect trackingDefect tracking
Test module DevelopmentTest module Development
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2828
What Makes Testing So Difficult?What Makes Testing So Difficult?
- Inherent Complexity of software- Inherent Complexity of software
- ill defined test base- ill defined test base
- Constructing an operational environment for testing purpose- Constructing an operational environment for testing purpose
- Intractable nature of testing- Intractable nature of testing
- No royal road for testing- No royal road for testing
- Idiosyncrasy of software- Idiosyncrasy of software
- Software continuously evolves- Software continuously evolves
- Testing under pressure- Testing under pressure
- Lake of test mind- Lake of test mind
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 2929
TestingTesting
소프트웨어 개발 비용소프트웨어 개발 비용
SpecificationSpecification11%11%
DesignDesign11%11%
CodingCoding13%13%
Testing & Testing & MaintenanceMaintenance
65%65%
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3030
Unit TestUnit Test
Unit Test Unit Test 란란 ??
소프트웨어 개발 시 하나의 소프트웨어 모듈이소프트웨어 개발 시 하나의 소프트웨어 모듈이
정상적으로 기능을 수행하는지 여부를 최소 수준의 정상적으로 기능을 수행하는지 여부를 최소 수준의
소스코드를 대상으로 시험하고 추가나 변경이 소스코드를 대상으로 시험하고 추가나 변경이
기존 소스의 영향을 찾아 개발기존 소스의 영향을 찾아 개발 , , 추가추가 , , 변경 시 조기에 변경 시 조기에
소프트웨어 버그를 찾아 수정한다소프트웨어 버그를 찾아 수정한다 ..
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3131
Regression TestRegression Test
Unit TestUnit Test
Unit TestUnit Test
Unit TestUnit Test
Regression TestRegression Test
Unit TestUnit Test
Unit TestUnit Test
Unit TestUnit Test
Regression TestRegression Test
Integration TestIntegration Test
Integration TestIntegration Test
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3232
Black Box & White Box TestBlack Box & White Box Test
SelectedSelectedinputsinputs
Black Box TestingBlack Box Testing
ResultantResultantoutputsoutputs
DesiredDesiredoutputoutput
SelectedSelectedinputsinputs
ResultantResultantoutputsoutputs
DesiredDesiredoutputoutput
White BoxWhite Box TestingTesting
InternalInternalbehaviorbehavior
SoftwareSoftwaredesigndesign
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3333
Black Box & White Box TestBlack Box & White Box Test
Specification, Program and Test CasesSpecification, Program and Test Cases
SpecificationSpecification
Test CaseTest Case
ProgramProgram
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3434
Unit TestUnit Test
Specification Base Test (Black Box Test)Specification Base Test (Black Box Test)
- Partition Testing- Partition Testing
- Boundary Value Testing- Boundary Value Testing
- Random Testing- Random Testing
Code Base Test (White Box Test)Code Base Test (White Box Test)
- Statement Testing- Statement Testing
- Branch Testing- Branch Testing
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3535
Partition TestingPartition Testing
domaindomainselectionselection
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3636
Boundary Value TestingBoundary Value Testing
invalidinvalid
aa bb
validvalid invalidinvalid
minmin min+min+ nomnom max+max+ maxmaxmin-min- max+max+
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3737
Random TestingRandom Testing
selectionselection
selectionselection
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3838
Statement TestingStatement Testing
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 3939
Branch TestingBranch Testing
aa
bb
dd
cc
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4040
Load TestLoad Test
목적목적
- Application- Application 의 병목 구간을 찾아 튜닝 의 병목 구간을 찾아 튜닝 (( 단위 성능 시단위 성능 시
험험 ))
- - 목표성능에 부합 여부 판단 목표성능에 부합 여부 판단 (( 통합 성능 시험통합 성능 시험 ))
- - 임계 성능 측정을 통한 서버 용량 산정임계 성능 측정을 통한 서버 용량 산정 및 확장 계획 및 확장 계획
(( 임계 성능 시험임계 성능 시험 ))
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4141
Concept for Load TestConcept for Load Test
- Named User, Concurrent User- Named User, Concurrent User
- Load- Load : Peak Load, Heavy Load, Average Load: Peak Load, Heavy Load, Average Load
- Response Time : Round Trip Time, Processing - Response Time : Round Trip Time, Processing
Time, Queuing TimeTime, Queuing Time
- Think Time- Think Time
- Request Interval- Request Interval
- Throughput : TPS, TPM- Throughput : TPS, TPM
Named UserNamed UserConcurrent UserConcurrent User
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4242
User and Request IntervalUser and Request Interval
비접속자비접속자 대기자대기자 사용자사용자
RequestRequestResponseResponse
Response TimeResponse TimeThink TimeThink Time
Request IntervalRequest Interval
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4343
ThroughputThroughputThro
ughput
Thro
ughput
(TPS)
(TPS)
User Count (Think Time = 0)User Count (Think Time = 0)
임계점임계점
임계 사용자 수임계 사용자 수저부하 구간저부하 구간
고부하 구간고부하 구간
경합 구간경합 구간
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4444
Response Time and TPSResponse Time and TPSThro
ughput
Thro
ughput
(TPS)
(TPS)
Concurrent UserConcurrent User
최대 허용 동시 사용자최대 허용 동시 사용자
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4545
Kind of Performance TestingKind of Performance Testing
목적목적
- - 단위 성능 시험 단위 성능 시험 : : 개개 업무의 최대 성능을 산출개개 업무의 최대 성능을 산출
- - 통합 성능 시험 통합 성능 시험 : : 업무 가중치와 부하 시나리오를 이용한 시험 업무 가중치와 부하 시나리오를 이용한 시험
- - 임계 성능 시험 임계 성능 시험 : : 시스템이 발휘할 수 있는 최대 성능 측정시스템이 발휘할 수 있는 최대 성능 측정
임계 성능 비율 임계 성능 비율 = = 호출빈도호출빈도 (Log (Log 분석분석 )/)/ 임계임계 TPS (TPS ( 단위성능시험단위성능시험 ))
단위 단위 APPAPP 별 별 TPS = TPS = 전체 전체 TPS TPS × × 단위 단위 AppApp 별 업무 가중치별 업무 가중치
단위 단위 APPAPP 별 별 TPS = TPS = 전체 전체 TPS (TPS ( 가중치 가중치 100%)100%)
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4646
Kind of Performance TestingKind of Performance Testing
방법방법
- Loop Back - Loop Back 시험 시험 : Loop Back Code: Loop Back Code 를 추가하여 시험를 추가하여 시험
- Tier - Tier 시험 시험 : Tier: Tier 간의 통신을 재현하여 측정간의 통신을 재현하여 측정
- Spike - Spike 시험 시험 : : 동시 사용자가 동시에 동시 사용자가 동시에 TransactionTransaction 을 발생을 발생 ..
- Extensibility - Extensibility 시험 시험 : : 시스템 증설에 대한 성능 향상 비율 측정시스템 증설에 대한 성능 향상 비율 측정 ..
- Availability - Availability 시험 시험 : Long run: Long run
성능확장 계수 성능확장 계수 (( 시스템 확장과 성능과의 관계시스템 확장과 성능과의 관계 ))
= = 성능 확장 비율 성능 확장 비율 / / 시스템 확장 비율시스템 확장 비율
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4747
Visual Studio Team SystemVisual Studio Team System
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4848
Static AnalysisStatic Analysis
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 4949
Static AnalysisStatic Analysis
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5050
Code CoverageCode Coverage
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5151
Test TypesTest Types
VSTSVSTS 에서 제공되는 테스트에서 제공되는 테스트
- Unit Test- Unit Test
- Manual Test- Manual Test
- Generic Test- Generic Test
- Web Test- Web Test
- Load Test- Load Test
- Ordered Test- Ordered Test
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5252
Unit Test MethodUnit Test Method
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5353
Unit Test ResultsUnit Test Results
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5454
Unit Testing With VSTSUnit Testing With VSTS
Unit Testing DemoUnit Testing Demo
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5555
Generic Test MethodGeneric Test Method
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5656
Generic Testing With VSTSGeneric Testing With VSTS
Generic Testing DemoGeneric Testing Demo
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5757
Web Test MethodWeb Test Method
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5858
Web Test Verify RulesWeb Test Verify Rules
유효성 검사 규칙유효성 검사 규칙 - - 텍스트 검색 텍스트 검색 ::
응답이 특정 텍스트를 포함하고 있는지를 응답이 특정 텍스트를 포함하고 있는지를 (( 또는 포함하지 않는지또는 포함하지 않는지를를 )) 검사검사 , , 정규식 및 대소문자 구분을 지원정규식 및 대소문자 구분을 지원
- - 최대 요청 시간 최대 요청 시간 : :
지정된 시간 이내에 응답을 받았는지를 검증지정된 시간 이내에 응답을 받았는지를 검증
- - 필요한 특성 값 필요한 특성 값 : :
태그의 특성이 특정 값인지 검증태그의 특성이 특정 값인지 검증 , , 대소문자 구분을 지원대소문자 구분을 지원
- - 필요한 태그 필요한 태그 : :
특정 태그가 존재하는지 검증 특정 태그가 존재하는지 검증 ; ; 최소 발생 값도 지원최소 발생 값도 지원
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 5959
Web Test VerifyWeb Test Verify
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6060
Web Test VerifyWeb Test Verify
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6161
Web Testing With VSTSWeb Testing With VSTS
Web Testing DemoWeb Testing Demo
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6262
Load Test MethodLoad Test Method
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6363
Load Test MethodLoad Test Method
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6464
Load Testing With VSTSLoad Testing With VSTS
Load Testing DemoLoad Testing Demo
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6565
ReferenceReference
- Microsoft, CODE COMPLETE Second Edition, 2004 Microsoft, CODE COMPLETE Second Edition, 2004
- WILEY, PATTERN-ORIENTED SOFTWARE ARCHITECTURE Vol1, 1996WILEY, PATTERN-ORIENTED SOFTWARE ARCHITECTURE Vol1, 1996
- http://www.microsoft.com/korea/eseminar/default.aspx?CateID=7 http://www.microsoft.com/korea/eseminar/default.aspx?CateID=7
- http://msdn2.microsoft.com/en-us/teamsystem/default.aspxhttp://msdn2.microsoft.com/en-us/teamsystem/default.aspx
- Microsoft, VISUAL STUDIO 2005 TEAM SYSTEM, 2005Microsoft, VISUAL STUDIO 2005 TEAM SYSTEM, 2005
04/21/2304/21/23 Devpia A&D EVADevpia A&D EVA 6666
감사합니감사합니다다 ..