test automation as a service (taaas) · pdf filetest automation as a service (taaas) - slide 1...
Post on 22-Mar-2018
217 Views
Preview:
TRANSCRIPT
Test Automation as a Service
(TaaaS) Next Generation Testing:
Innovations and Challenges Jonathon Lee Wright
Director of Testing
jlwright@iee.org
@jonathon_wright
www.TaaaS.net (blog)
www.automation.org.uk
automation.sharepoint.com
www.linkedin.com/in/automation
18th May 2011 Test Automation as a Service (TaaaS) - Slide 1
Introduction
Agenda
• Why the cloud?
• Automation in the cloud;
• New opportunities and benefits;
• New risks;
• What we are doing now;
• The future;
• Live demo.
Test Automation as a Service (TaaaS) - Slide 2
Introduction
18th May 2011
Why the cloud?
• The cloud provides no up-front investment,
featuring instant scalability, flexibility and availability;
• Pay as you use (PAYU), no more expensive vendor licencing
costs or un-utilised maintenance agreements;
• Complex test environments on demand:
“built to expect & withstand failure”;
• Established technology avoiding support issues with non-
commercial products;
• Avoid the minefield of complexity requiring internal specialist
staff skills following industry best practice;
• 100% web delivered providing collaboration and mobility.
Test Automation as a Service (TaaaS) - Slide 3
Cloud Services
18th May 2011
New Testing Opportunities
• Cross Platform and Compatibility - browser, operating system,
mobile devices, script runtimes, languages, HTML versions;
– Market Configuration Testing – 10+ Applications (Shop / ShelfReview / Nominations),
10+ Clients (Tesco / Macy‟s), 10+ Workstreams (Tesco UK / Tesco France),
10+ Languages (en-GB / fr-CA), 10+ Platforms (XP-SP1, SP2 & SP3),
10+ Browsers (I.E 7, 8, 9), 10+ Runtimes (Flash 9.0.44 ~ 10.3.181).
• Accessibility/Usability/UXD/WAI - due diligence in guaranteeing
accessibility to people with disabilities;
– 88% of the delegate organisations here today failed W3C compliance tests;
• Globalisation – realistic performance & load testing plus
compliance with the commercial and civil law of the country;
• Search Engine Optimisation – periodic ranking checks;
• Penetration, Performance & Load Testing.
Test Automation as a Service (TaaaS) - Slide 4
New Opportunities
18th May 2011
New Risks (in the cloud)
Test Automation as a Service (TaaaS) - Slide 5
New Risks
18th May 2011
Commercial Ecosystems Fulfilment Systems
• What happens when these Services FAIL?
– Systemic Failure? (e.g. DNS or Network link)
– Alternative service solution provider?
– Unique Services? (such as GPS)
Taking Automation to the Cloud
Test Automation as a Service (TaaaS) - Slide 6
Cloud Automation
18th May 2011
Customer Systems
Intranet
Site
Test Suite CMMI/ISO 9000
Rule StoreSoftware Build
Process
Desktop
Application Web Site
Web services PersistenceVirtual lab
The Cloud (System Infrastructure)
Customer Automation Suite
Business Level
Keyword Definition
United Reporting
Credentials
Flexible Cost
Planning
Support
Test Data
DashboardTest Asset
Management
Automation Lifecycle Management
(Test Automation as a Service Model)
Monitoring/
Operational management
Configuration
Reporting
Billing
Customer Take on Customer support
Development
DeploymentMaintenance
Why this approach?
Test Automation as a Service (TaaaS) - Slide 7
Cloud Automation
18th May 2011
Approach BPM BPT Keyword Data Modularity Tools
Traditional ○ ○ ○ ○ ● Vendor Tool (Market Leader)
Hybrid ● ● ● ● ● .NET Development
(Visual Studio 2010 + Coded UI APIs)
○ - Limited Support
● - Full Support
Traditional (Vendor Tool)
Advantages Constraints
Vendor Support
(maintenance subscription)
Object Repository
(object-based Record /
Replay)
Single Technology Stack
(SPOF)
Tool Vendor Dependency
C# Project incompatible with
Visual Studio
Technology Blocks
(Custom Flex Objects/Classes)
Limited non-technical tester
interface
Hybrid (.NET Development)
Advantages Constraints
Multiple Technology Stack
(VS2010 / C#.net / Silverlight /
Azure / SQL / WCF /
CodedUI / Ranorex API /
XML / XPATH / TFS)
Integration with development
(Visual Studio 2010)
Integration with deployment
(TeamCity)
Integration with testing
(Microsoft Test Manager)
In-house framework
development & maintenance
effort.
Additional skills / knowledge
resources
Object Repository
(limited Record / Replay
support)
What is different with this approach?
Test Automation as a Service (TaaaS) - Slide 8
Cloud Automation
18th May 2011
• Design tests before release ‘first day’ automation testing – Create complex test scenarios before the application has even been written;
• The test language is self-validating and human readable – The verbs and nouns in the domain specific language are written in natural
language which uses context sensitive validation;
A1
B1
Login.Process
Buy.Tickets
Check.Tickets
My.Account
Logout.Process
D1
C1
B2B4
C3B3C2
B5
D3C4 D2
D4 D5
C5
Lo
gin
.Pro
cess
Action Object Event
Enter Username Text
Enter Password Text
Press Login Button
Nouns
Verbs Generic Reusable Libraries
Act
ions
Obj
ect
Han
dler
Eve
nt
Han
dler
Press | Login | Button Example Business Level Keyword
BPM
What is different with this approach?
Test Automation as a Service (TaaaS) - Slide 9
Cloud Automation
18th May 2011
• The test definition is not implementation specific – Test execution is performed by cloud based technology adapters which act as
interpreters for the vendor agnostic and implementation agnostic language;
Test Run Generate Environment
Under Test (EUT) Select Platform
Adapter Select Test
Adapter Select Technology
Adapter
• Application Server
• Database Server
• End-Client Machine
• Windows 2008 RC1
• Windows 2003 RC2
• Windows 7 SP1
• Load Testing
• Performance
• Functional
• Coded-UI (Win32)
• Selenium (ASP.net)
• Ranorex (Flex 4.0)
• The definition of a test artefacts is self-maintaining – Asset loader uses fuzzy logic matching to identify UI objects;
Enter password here ……
Please enter your password to logon
logon logoff
Enter password here ……
Please enter your password to login
login logoff
1. XPATH Check
2. Regular Expressions
3. Descriptive Programming
4. Fuzzy Logic
5. Artificial Intelligence
n/a
Event Based / BDD Testing
Test Automation as a Service (TaaaS) - Slide 10
Cloud Automation
18th May 2011
• Automated testing based on trigger events (passive & active): – Third party system events such as user story or defect status workflow change;
‒ Continuous build and integration or source code gated check-in.
Passiv
e
Acti
ve
Where are we now?
Test Automation as a Service (TaaaS) - Slide 11
Cloud Automation
18th May 2011
Development (.NET 4.0)
AutomationDatabase
Automation Portal (TaaaS)
Application Under Test
2003/2007/2010
Lab Management
Windows
Service
ObjectHandler.dll
DataHandler.dll
EventHandler.dll
Application Programming Interface
Unified Reporting Model
DashboardReportingTest Management Tool Team Foundation Server
Application Lifecycle Management
Test Automation as a Service (TaaaS)
Cloud Web Endpoint
Test Scheduler Service
How it works (in the cloud)
Test Automation as a Service (TaaaS) - Slide 12
Cloud Automation
18th May 2011
End Users
QA Person
Run a test«uses»
Schedule a future
test
«uses»
TaaaS Portal
Silverlight
Dashboard
Test Runner Environment - A Virtual Machine
created by the Framework VM Dispenser
TestRunner
Test Data
Set
Technology
AdapterTest Adapter
Platform
Adapter
Client System Test Repository
Business
Level
Keywords
(BPM/BPT)
Test Data
SetsResults
Object
Repository
View real time
test status
«uses»
Business User
«uses»
View historic test
results
«uses»
Dashboard Service
Test Assets Repository
(MTM / Quality Centre)
TD-API
Adapter
Results Service
Test Management
Unified Reporting (Dashboard)Check Development
Progress
Framework
Results
Adapter
Silverlight
Dashboard
Developer
View Live Defect
Build Client System
Build Automation
(Team City / TFS)
Framework
Object
Adapter
Asset Scanner Service
Benefits to the Customer
Test Automation as a Service (TaaaS) - Slide 13
Cloud Automation
18th May 2011
On Demand
No capital
Investment
Minimal Time
Investment
Cost
Controlled
Public facing Web sites
Source code
Public facing Web Services
Test Assets
Intranet
Desktop applications
Business Process Model
Test System
Software Build Process
Customer Assets
Domain Experts
Use Existing
Test Assets
TaaaS
Load Testing
Penetration Testing
ISO/CMMI Progress
Contribution
Market Compliance
Search Engine Optimisation
Code Standards
Regression Testing
Accessibility Standards
Performance Testing
Customer Defined Tests
‘Keep it Simple’ Automation
Test Automation as a Service (TaaaS) - Slide 14
Cloud Automation
18th May 2011
User stories
represented by a
number of Business
Level Keywords
Save Test (Selectable
in the future from the
Your Saved Tests List
Box)
Start Test (As per
the custom list of
business level
keywords)
TaaaS vs. DIY
Test Automation as a Service (TaaaS) - Slide 15
Cloud Automation
18th May 2011
TaaaS
Future forecast "Cloudy"
1. Avoid the Hype: – Ensure that the critical business flow can be automated followed by
complex custom components "avoid automating the simple components
first“ e.g. simple login pages;
2. Don't be tempted: – By third generation automation vendors offering so-called "script-less"
tools. Be realistic, you need to be able support custom classes and the
latest technologies without having to wait on the possibility that it may be
supported in the next release;
3. Integration into testing process: – Automation is no longer an "us and them" scenario between manual and
automated tests;
4. Make the necessary plans: – How easy is it to start using the service and how easy is it to move away
from the service.
Test Automation as a Service (TaaaS) - Slide 16
The Future
18th May 2011
Something to think about
“The best aspects of these proven test approaches demonstrate
how they have evolved over the past decade, and this echoes
some of the changes towards more lean and agile business
methodologies. They are in a constant state of evolution – just
as the underpinning technology evolves over time.”
my section in „Experiences in Test Automation’ book
Dorothy Graham & Mark Fewster 2011
Test Automation as a Service (TaaaS) - Slide 17
The Future
18th May 2011
Automation - Demo
Test Automation as a Service (TaaaS) - Slide 18
Cloud Automation
18th May 2011
top related