windows programming introduction to test monkey & cr toolswkchen/courses/wprog/... · •...

1

NTUT CSIE

Windows ProgrammingWindows ProgrammingWindows Programming

陳偉凱陳偉凱台北科大資工系台北科大資工系

Introduction toIntroduction toTest Monkey & CR toolsTest Monkey & CR tools

2NTUT CSIE

GUI Testing Using Test MonkeysGUI Testing Using Test Monkeys1/41/4

•• Test monkeyTest monkey–– A A fully automatedfully automated testing tooltesting tool–– DoesnDoesn’’t know how to use AUT.t know how to use AUT.–– Performs mouse clicks (events) and key Performs mouse clicks (events) and key

stokes randomly (and quickly)stokes randomly (and quickly)–– BlackBlack--boxbox testingtesting

•• Why use test monkeys?Why use test monkeys?–– There are There are bugs undetectedbugs undetected by either by either

automatic or manual GUI testing.automatic or manual GUI testing.–– Test monkeys are often used Test monkeys are often used in additionin addition to to

manual and automatic tests.manual and automatic tests.

2

3NTUT CSIE


•• DumbDumb test monkeystest monkeys–– Know nothing about AUTKnow nothing about AUT–– CanCan’’t recognize a bug when they see onet recognize a bug when they see one

•• SmartSmart test monkeystest monkeys–– Generate random actions from some knowledge Generate random actions from some knowledge

•• Knowledge from state table or model of the softwareKnowledge from state table or model of the software•• Traverse state model to move from one state to anotherTraverse state model to move from one state to another

•• Do test monkeys find bugs?Do test monkeys find bugs?–– Testers report Testers report finding serious bugsfinding serious bugs by using test by using test

monkeysmonkeys–– Microsoft reported Microsoft reported 10%10%--20%20% percent of bugs are percent of bugs are

found by test monkeysfound by test monkeys•• Click randomly on an empty Excel overnight crashes it.Click randomly on an empty Excel overnight crashes it.

4NTUT CSIE


•• Benefits of using test monkeysBenefits of using test monkeys–– Randomly find nasty bugs that other means could notRandomly find nasty bugs that other means could not

•• Can be used at Can be used at any stageany stage during development cycleduring development cycle•• Can be used to test Can be used to test different projectsdifferent projects•• Change of GUI does not affect the performance of test Change of GUI does not affect the performance of test

monkeysmonkeys–– Will run until they crash the systemWill run until they crash the system

•• Find memory and resource leaks effectivelyFind memory and resource leaks effectively•• Increase confidenceIncrease confidence if they run for days without failureif they run for days without failure

–– CoverCover gapsgaps between manual and automated testingbetween manual and automated testing•• If they find bugs If they find bugs revise test plan revise test plan

–– Find the Find the most unwanted bugsmost unwanted bugs (hang or crash)(hang or crash)

3

5NTUT CSIE


•• How to monitor test monkeysHow to monitor test monkeys–– Log Log the actions performed by test monkeysthe actions performed by test monkeys–– Run a test monkey with a Run a test monkey with a debuggerdebugger

•• When the monkey encounter a bug, the debugger halts the When the monkey encounter a bug, the debugger halts the monkeymonkey

–– Use Use video camerasvideo cameras to record the monkeyto record the monkey’’s actionss actions•• Testers can fast forward the tape to reproduce the bugTesters can fast forward the tape to reproduce the bug

6NTUT CSIE

GUI Capture and Replay ToolsGUI Capture and Replay Tools1/131/13

•• Capture/Replay toolsCapture/Replay tools–– Currently, the Currently, the most most

popularpopular GUI testing GUI testing methodmethod

•• How it works?How it works?–– A CRA CR--tool tool capturecapture all all

manual user manual user interactions on AUT interactions on AUT into a test scriptinto a test script

–– The test script can be The test script can be replayedreplayed to generate to generate user interactionsuser interactions

–– Useful for Useful for regression regression testingtesting

Test Script

CR-ToolTester

Application under test

interact

editing

C/R

R/W

4

7NTUT CSIE


•• Common CRCommon CR--tool functionstool functions–– Capture modeCapture mode

•• Component awareComponent aware: recognize each GUI element and capture : recognize each GUI element and capture all object characteristics (name, color, label, value)all object characteristics (name, color, label, value)

–– ProgrammingProgramming•• The captured user interactions are stored in a The captured user interactions are stored in a test scripttest script (e.g. (e.g.

C, VB, TSL). C, VB, TSL). –– CheckpointsCheckpoints

•• Tester can insert checkpoints in the test script. Tester can insert checkpoints in the test script. •• Both view of windowsBoth view of windows--objects (color, size, etc.) and functional objects (color, size, etc.) and functional

characteristics (model) of the AUT can be verifiedcharacteristics (model) of the AUT can be verified–– Replay modeReplay mode

•• Test scripts can be replayed at any time.Test scripts can be replayed at any time.•• Permits GUI replay, even if the GUI has been modified by a Permits GUI replay, even if the GUI has been modified by a

change in the software. change in the software. •• If checkpoints are violated, then the test fails.If checkpoints are violated, then the test fails.

8NTUT CSIE


•• Commercial CR toolsCommercial CR tools11

–– IBM Rational Robot ($4240)IBM Rational Robot ($4240)•• Works with MS Visual Studio 6.Works with MS Visual Studio 6.•• Supports capture/replaySupports capture/replay•• Test script: Test script: SQABasicSQABasic (similar to VB)(similar to VB)

–– IBM Rational Visual TestIBM Rational Visual Test•• Works with MS Visual Studio 6.Works with MS Visual Studio 6.•• Supports capture/replaySupports capture/replay•• Test script: Test script: TestBasicTestBasic

–– SilkTestSilkTest (Segue)(Segue)•• Works with MFC and extends to nonWorks with MFC and extends to non--MFC componentsMFC components•• Supports capture/replaySupports capture/replay•• Test script: 4Test.Test script: 4Test.•• Provides wizards to check bitmaps and verification statementsProvides wizards to check bitmaps and verification statements

5

9NTUT CSIE


•• Commercial CR ToolsCommercial CR Tools22

–– WinrunnerWinrunner (Mercury)(Mercury)•• Supports capture/replaySupports capture/replay•• Works for Windows, Web, and terminal emulationWorks for Windows, Web, and terminal emulation•• Selectable recording based on object or coordinatesSelectable recording based on object or coordinates•• Test script: TSLTest script: TSL

–– QuickTestQuickTest (Mercury)(Mercury)–– LoadRunner (Mercury Interactive)LoadRunner (Mercury Interactive)

•• Use test scripts (e.g. Winrunner) to perform actions a real Use test scripts (e.g. Winrunner) to perform actions a real person (virtual user) would doperson (virtual user) would do

•• Conducts load or stress testingConducts load or stress testing•• Can simulate thousands of virtual users to find system Can simulate thousands of virtual users to find system

bottlenecksbottlenecks

10NTUT CSIE


•• Open Source CR toolsOpen Source CR tools–– AbbotAbbot

•• http://http://abbot.sourceforge.netabbot.sourceforge.net•• Works with Java swing/AWT applicationsWorks with Java swing/AWT applications•• Supports capture/replaySupports capture/replay•• Provides a visual test script editing toolProvides a visual test script editing tool•• Test scripts: XML or javaTest scripts: XML or java

–– GTT GTT •• http://http://www.ntut.edu.tw/~wkchen/gttwww.ntut.edu.tw/~wkchen/gtt//•• Works with Java swing applicationsWorks with Java swing applications•• Supports capture/replaySupports capture/replay•• Provides a visual test script editing toolProvides a visual test script editing tool•• TDD is possibleTDD is possible•• Test scripts: XMLTest scripts: XML

6

11NTUT CSIE

CRCR--tool exercisetool exercise

12NTUT CSIE


•• TrapsTraps when Accessing GUIwhen Accessing GUI--ObjectsObjects–– Make sure your CRMake sure your CR--Tools Tools recognize GUI objectsrecognize GUI objects and and

also realso re--recognize them if the GUI layout changed.recognize them if the GUI layout changed.–– Be careful: A CRBe careful: A CR--tool recognize VC++ GUI objects tool recognize VC++ GUI objects

may not recognize objects with other applicationsmay not recognize objects with other applications–– GUI objects are recognized by their ID or a set of GUI objects are recognized by their ID or a set of

attribute values that uniquely identify themattribute values that uniquely identify them•• Sensitive to changes of these values.Sensitive to changes of these values.•• Visual C++ sometimes change ID's of GUIVisual C++ sometimes change ID's of GUI--objects. The CRobjects. The CR--

Tool must be manually reTool must be manually re--taught about all changed GUI taught about all changed GUI objects. objects.

•• Problem occurs if GUI objects are eliminated or redesigned Problem occurs if GUI objects are eliminated or redesigned due to functional changes in the new product release. due to functional changes in the new product release.

7

13NTUT CSIE


•• Traps when Testing FunctionalityTraps when Testing Functionality–– CapturingCapturing interactions interactions does not testdoes not test anythinganything

•• To test if the data is saved on the server, you have to look forTo test if the data is saved on the server, you have to look forthe data on the serverthe data on the server

•• CRCR--tools generally do not offer this kinds of checkpointtools generally do not offer this kinds of checkpoint–– AppearancesAppearances of GUI objects depends on current stateof GUI objects depends on current state

•• Buttons are enabled/disabled depending on fieldButtons are enabled/disabled depending on field--values, values, •• Data fields change color to gray and become readData fields change color to gray and become read--only, only, •• Toolbars and menu entries change during operation, Toolbars and menu entries change during operation,

–– The bitmap you stored as a reference bitmap contains The bitmap you stored as a reference bitmap contains system system clock clock output output

–– Another application eats up windows Another application eats up windows resourcesresources–– An email An email message popping upmessage popping up captures the focus captures the focus –– The AUT The AUT iconifiesiconifies–– Other Other surprisessurprises

14NTUT CSIE


•• Traps of Style Guide Testing Traps of Style Guide Testing –– Style guideStyle guide

•• Consistent look and feelConsistent look and feel–– OK and Cancel are always located on the same placeOK and Cancel are always located on the same place–– All fields must be perfectly alignedAll fields must be perfectly aligned–– Texts must be legible and displayed with sufficient contrastTexts must be legible and displayed with sufficient contrast–– A contextA context--sensitive help must be accessible for each button.sensitive help must be accessible for each button.

–– Style guide must be applied for entire applicationStyle guide must be applied for entire application•• Automated style guide testing is beneficialAutomated style guide testing is beneficial•• Can be done along with automated GUI testing without Can be done along with automated GUI testing without

overheadoverhead–– HoweverHowever

•• Style guide criteria are difficult to formulate and quantifyStyle guide criteria are difficult to formulate and quantify•• Test automation without formulation always failsTest automation without formulation always fails

8

15NTUT CSIE


•• GUI test automation is a programming taskGUI test automation is a programming task–– Capturing only helps to get a Capturing only helps to get a prototypeprototype of a test caseof a test case–– Captured scripts need additional Captured scripts need additional programmingprogramming to to

obtain useful checkpointsobtain useful checkpoints–– Captured scripts require Captured scripts require maintenancemaintenance to keep them to keep them

reusablereusable•• GUI testing is a software GUI testing is a software specificationspecification tasktask

–– The The test casestest cases must be defined before capturingmust be defined before capturing–– The specification must be much more The specification must be much more detaileddetailed than than

for manual testing.for manual testing.–– To be successful: planning to automate GUI testing To be successful: planning to automate GUI testing

must already have a must already have a testing processtesting process in placein place

16NTUT CSIE


•• A GUI Testing Process of the A GUI Testing Process of the imbusimbus companycompany–– test planning test planning test specification test specification

test implementation test implementation automated testing. automated testing.

Courtesy of [5]

9

17NTUT CSIE


•• Measurement of Measurement of ExpendituresExpenditures–– AutomatedAutomated GUI testing GUI testing manualmanual GUI testingGUI testing–– Measurement Measurement targettarget softwaresoftware

•• A PC A PC tooltool for GSM radio base stationsfor GSM radio base stations•• GUIGUI for commissioning, parameterization, hardware for commissioning, parameterization, hardware

diagnostics, firmware downloading, equipment data base diagnostics, firmware downloading, equipment data base creation, increation, in--field and offfield and off--line diagnostics of (different) base line diagnostics of (different) base stations, and a fullstations, and a full--graphics editor for equipment definitiongraphics editor for equipment definition

•• Developed using MS Developed using MS VC++VC++/Visual Studio/Visual Studio•• About About 100,000 lines100,000 lines of C++ codeof C++ code

18NTUT CSIE

Courtesy of [5]


Break even of GUI test automationBreak even of GUI test automation

Manual(hours)

Automatic(hours)

Manual(hours)

Time to interpret testing results

Break even point5 runs

(auto hours / manual hours)

10

19NTUT CSIE


•• Observations on ExpendituresObservations on Expenditures–– BreakBreak--even pointeven point 2.02.0

•• Assuming Assuming no further modificationsno further modifications of test scripts is required of test scripts is required for the 2nd run of automated GUI testing (almost impossible)for the 2nd run of automated GUI testing (almost impossible)

•• Assuming Assuming no human errorno human error for manual GUI testing (unlikely)for manual GUI testing (unlikely)

–– Buy and use a CRBuy and use a CR--tool tool •• Test cost Test cost increaseincrease to 125%to 125%--150%150%•• Additional cost due to trapsAdditional cost due to traps

–– Use CRUse CR--tool with a testing framework tool with a testing framework •• Use CRUse CR--tool as a tool as a cornerstonecornerstone, not a solution, not a solution•• Cost down to 40%Cost down to 40% (typical product test cycle 10)(typical product test cycle 10)

NTUT CSIE

A tutorial of AbbotA tutorial of Abbot

11

21NTUT CSIE

Costello quick startCostello quick start1/31/3

1.1. Launch the CostelloLaunch the Costello2.2. Create a new scriptCreate a new script

•• FileFile-->New Script>New Script3.3. Launch the GUILaunch the GUI

•• TestTest-->Launch>Launch4.4. Record some user actionsRecord some user actions

•• CaptureCapture-->All Actions >All Actions 5.5. Add some assertionsAdd some assertions6.6. Run the scriptRun the script

Class name

Class path

22NTUT CSIE

Costello Demo I: CalculatorCostello Demo I: Calculator2/32/3

•• 錄影錄影( CE 123+321=444) ( CE 123+321=444) 綠燈綠燈•• Slow playbackSlow playback 綠燈綠燈•• Assertion 444 Assertion 444 綠燈綠燈•• EditingEditing

–– 改改Sequence name: Sequence name: testAddtestAdd–– 刪除刪除event 1event 1 紅燈紅燈–– 錄影補回錄影補回event 1event 1 綠燈綠燈

•• Save scriptSave script•• Insert script Insert script

–– 保留保留testAddtestAdd sequence sequence –– 改名為改名為testSubtestSub–– 刪除刪除event addevent add，錄影加上，錄影加上

subsub 紅燈紅燈–– 更改更改 assertion assertion --198198 綠燈綠燈

•• Insert annotationInsert annotation–– ““start start testAddtestAdd”” –– require user require user

dismissdismiss–– ““start start testSubtestSub””

•• Insert commentInsert comment–– ““這是加法的測試這是加法的測試””–– ““這是減法的測試這是減法的測試””

•• Layout changed (1,2,3 Layout changed (1,2,3 3,1,2) 3,1,2) Test Launch (Test Launch (ctrl+Lctrl+L)) 綠燈綠燈

•• AUT changed (eliminate 1)AUT changed (eliminate 1)–– 修改修改 testSubtestSub 的的AUTAUT為為

LayoutChangeLayoutChange版本版本–– Test Launch (Test Launch (ctrl+Lctrl+L) )

•• testAddtestAdd Timeout Timeout 紅燈紅燈•• testSubtestSub 綠燈綠燈

1. 錄影/停止錄影: shift + f22. 找component: 滑鼠移到component上，再按 shift+f1

12

23NTUT CSIE

Costello Demo II: ClassEditorCostello Demo II: ClassEditor3/33/3

•• http://classeditor.sourceforge.net/http://classeditor.sourceforge.net/

windows programming introduction to test monkey & cr toolswkchen/courses/wprog/... · •...

Documents