etsi update on ttcn-3 language evolution and tool conformance testing nikolay pakulin contributors:...

ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

Nikolay Pakulin

Contributors:

Ina Schieferdecker

Jens Grabowski

TTCN-3 User Conference2012, Bangelore, India

Agenda

STF 430: TTCN3 Evolution TTCN3 Overview Advanced TRI Continuous Signals

STF 433: TTCN3 Tool Conformance Testing Motivation Methodology Challenges Requirements-driven Test Development

Slide 2ETSI Update on TTCN-3 Language Evolution and Tool Conformance Testing

TTCN-3 Evolution


Nikolay Pakulin

on behalf of STF430


STF administrative data

STF 430: April 2011 – December 2011 (78 mDays)

Members: Gyorgy Rethy, Ericsson Jens Grabowski, University of Goettingen Ina Schieferdecker, Fraunhofer FOKUS Jacob Wieland, Testing Technologies

STF 430 sessions in 2011 Session 1: 24-27 May Session 2: 27 June - 1 July Session 3: 26 - 30 September Session 4: 28 November - 2 December



IDL XSDLanguagemappings ASN.1

Extensions

Advancedparameteri-

zation

Behaviour

typesStatic

configurationDocumentation

t3doc

TTCN-3 Structuring:

Imports, Groups, Attributes

Core

language

TTCN-3 Technology Overview

TTCN-3 Behaviour

TTCN-3 Data

C C++TRI/TCI

mapping Java XML C#

Real-time

support

Continuoussignalsupport

AdvancedTRI



TTCN-3 language evolution

Tasks Resolution and implementation of CRs

Prepare the drafts for ETSI publication:Parts 1 and 4 to 10 of the ES 201 873 seriesThe 4 extension packages (ES 202 781, ~782, ~784,

~785)

Cooperation with STF 160 on language requirements of LTE testing

Two new language extension packages Advanced TRI Continuous signal support



The version forest Approved new versions

TTCN-3 Part-1: Core Language V4.4.1 (44 CRs) TTCN-3 Part-4: Operational Semantics V4.4.1 (1 CR) TTCN-3 Part-5: TTCN-3 Runtime Interface V4.4.1 (2 CRs) TTCN-3 Part-6: TTCN-3 Control Interface V4.4.1 (8 CRs) TTCN-3 Part-7: Using ASN.1 with TTCN-3 V4.4.1 (1 CR) TTCN-3 Part-8: Using IDL with TTCN-3 V4.4.1 (2 CR) TTCN-3 Part-9: Using XML with TTCN-3 V4.4.1 (12 CRs) TTCN-3 Part-10: TTCN-3 docum. tags V4.4.1 (1 CRs) TTCN-3 Extension: Behaviour Types V1.2.1 (3 CRs) TTCN-3 Extension: Configuration and deployment V1.2.1 (1 CR) TTCN-3 Extension: Real-Time and Performance Testing V1.2.2 (2 CRs)

Not republished: TTCN-3 Extension: Advanced Parameterization V1.3.1

New: TTCN-3 Extension: Advanced TRI V1.1.1 TTCN-3 Extension: Continuous signal support V1.1.1



New features in Part 1

5417: Support of parametrized map/unmap

var MyConfigType cfg:={ option := 1, lock := false};

...map(mtc:Port4, system:PCO2) param (cfg);

5262: Partially constrained structured types 5938: Type restriction by template list

type MyRecord MyRecordSub5 ( {f1 := *, f2 := "user", f3 := pattern "password|Password" }, { f1 := (1 .. 10), f2 := "User“, f3 := ? })

5937: Generalized annotation of attributes to declarations/members



New: Advanced TRI 1(3)

Historically, TTCN has been used to test communication protocols which typically use encoded messages. This has been reflected in the TRI SA and TCI CD design of TTCN-3 by encoding and decoding messages to/from bitstrings.

However, TTCN-3 also supports signature-based communication for which the transformation of objects into bitstrings and vice versa is cumbersome.

Therefore, an alternative API is being proposed along which TTCN-3 values can be directly passed to/from the SUT.

Old and new TRI can be used in combination!




Redefinition of TRI Data Interface TriMessageType use Value TriAddressType use Value TriAddressListType use ValueList TriSignatureIdType unchanged TriParameterType unchanged (not being

used) TriParameterPassingModeType

unchanged (not being used)

TriParameterListType TciParameterListType (by that, the TciParameterPassingMode and Values for parameters will be used)

TriExceptionType use Value




Changes to Message-based communication operations triSend xtriSend triSendBC xtriSendBC triSendMC xtriSendMC triEnqueueMsg xtriEnqueueMsg

New Value xtriConvert(in any value, in Type typeHypothesis)

This operation shall be called whenever the TE has to convert a value. The TE might convert immediately after reception of the value, or might for performance considerations postpone the conversion until the actual access to the value.

Changes to Procedure-based communication operations triCall xtriCall etc.

Changes to Miscellaneous operations triExternalFunction xtriExternalFunction



TTCN-3 Extension: Continuous signal support 1(4)

This new package defines concepts for testing systems using continuous signals.

The package introduces notions of time and sampling, notions of streams, stream ports and stream variables,

and definitions allowing an automaton alike control flow

structure supporting the specification of hybrid behavior (i.e., combined definition of discrete and continuous behaviors).




The following items shall give an impression about the contents of the package. Details and further operations can be found in the package definition.

Concepts for time and sampling: access to global time: now operator Setting stepsize for sampling: stepsize operator

Streams New port type for streams, e.g.,:

type port StreamIn stream { in float } type port StreamOut stream { out float }

Operations for accessing data streams: retrieving or setting actual values: value operation retrieval of time related information: timestamp operation Retrieval of actual stepsize of a port: delta operation




Streams (cont.) navigation on streams

previous event: prev operation event at a certain point in time: at operation

stream extraction Retrieval of (partial) history: history operation

Modes define states in hybrid automata. Modes may

be composed sequential or parallel have entry and exit conditions have invariant parts have a duration be parameterisable etc.




15Slide 15ETSI Update on TTCN-3 Language Evolution and Tool

Conformance Testing

A Conformance Test Suite for TTCN-3 Tools

Bogdan Stanca-Kaposta, Testing Technologies

Andras Kovacs, Broadbit

Nikolay Pakulin, ISPRAS


Motivation

Many TTCN-3 compilers (~7 commercial, 2 internal) High interest in the language Complex standard: hundreds of pages No way to check the compliance of TTCN-3 tools Quality of a test tool is critical

Relevance: For users:

Is my tool standard compliant? Can I switch tools easily / vendor lock-in?

For tool vendors: Where can I improve the standard compliance of my

tool? Possibly later: compliance certification

For standard developers: Where is need for clarification?



STF 433

Specialist Task Force initiated by ETSI MTS Manpower: 3 persons for 100 days total Dates: November 2011 – Early 2012 Work is based on the existing framework.

Task: Update the existing TTCN3 ATS to the newer TTCN-3

revision Extend the conformance test suite for TTCN-3 tools.

Three deliverables: ETSI TS 102 950-1 V1.2.1 (2012-05) – ICS ETSI TS 102 950-2 V1.2.1 (2012-05) – TSS & TP ETSI TS 102 950-3 V1.2.1 (2012-05) – ATS & IXIT Approved by MTS in May 2012.

Downloadable from the ETSI download area right now!



Methodology (1/7)

Main steps:1. Validate existing tests against new revision of the

standard.

2. Develop conformance tests. Follow the priority list discussed with MTS and vendors.

3. Develop adapters for selected tools. Validate and fix tests.

4. Automate test execution.



Methodology (2/7)

Adjusting Conformance ATS from TTCN3 rev. 4.2.1 to rev. 4.3.1: Was: 733 tests, 130 pp. of test purposes

Validate that test purposes still hold Update modified test purposes Revoke outdated tests Update ATS folder structure

Approximately 30% of the STF 433 effort

New tests development Clause priority list:

Not all clauses can be covered. Selection was made: Vendor feedback + most basic functionality.

There is no perfect selection!



Methodology (3/7)

Development of conformance tests: The ATS is not the test automation, but the test input! In most cases: 1 module = 1 conformance test. Test automation provided for certain vendors

Elvior and Testing Technologies Test suite structure: organized according to clauses.

Expected output for verdict determination: @verdict pass accept/reject [expectedoutput]

Examples: @verdict pass reject @verdict pass accept, ttcn3verdict:pass @verdict pass accept, noexecution

Purpose specification: @purpose documentreference, description Documentreference: part:clause

Examples: @purpose 1:5, Ensure that when the IUT loads a module containing

some definitions before the module declaration then the module is rejected.



Methodology (4/7)

Development of conformance tests (cont.): Complete TTCN-3 test input example:

/*************************************************** ** @purpose 1:5.2.2, Ensure that the IUT correctly handles the uniqueness of variable names in its scope. ** @verdict pass accept, ttcn3verdict:pass ***************************************************/module Sem_050202_Uniqueness_001 {

import from Sem_050202_Uniqueness_001_import { const all;

}type component GeneralComp {}function f_funcScope() {

var boolean repeatedIdentifier := true;

if(repeatedIdentifier==true) { setverdict(pass); }

}testcase TC_Sem_050202_Uniqueness_001()

runs on GeneralComp { f_funcScope(); }

control { execute(TC_Sem_050202_Uniqueness_001()); }}

module Sem_050202_Uniqueness_001_import {// repeated indentifier // from imported module// is allowed

const integer repeatedIdentifier := 0;}



Methodology (5/7)

Creation of documentation: Framework already provided by STF393 (Proforma). Little time for documentation. Working on a single source saves time.

Solution: Write test purposes and expected tool outputs directly

into the test inputs. Generate tables for the documentation from the ATS:

Test purpose tables. Test suite structure.

Much easier to maintain! Documentation generation tools developed as part of

STF409.



Methodology (6/7)

Validation and fixing: Validation and fixing took more than half of the

development time!

STF internal validation with 2 tools: TTworkbench: compile-time and execution. TestCast: compile-time and execution.

Developed necessary adapters Loopback, proc. call

Lots of feedback from TTech and Elvior!



Methodology (7/7)

Executing the ATS:

ATSSem_060101_TopLevel_001.ttcnNegSem_060303_component_types_001.ttcn…

TTCN-3 Tool (SUT)

TTCN-3 FileTTCN-3 FileTTCN-3 FileSem_060101_TopLevel_001.ttcn

Tool output

Rejection as invalid Rejection to executeExecution with results

(e.g., pass/inconc/fail)

Tool output evaluation

** @verdict pass accept, ttcn3verdict:pass

failpass

Test inputExpected

output



Discussion

Compliance does not imply safety from a vendor lock-in! Testing is never complete! With v1.1.1 of the ATS, about 32% of the core standard clauses

are covered with at least one test. Ideally, we should have many tests for each clause.

Compliance means less likelihood for vendor lock-in. With better coverage and higher number of test cases, we

improve the quality of this statement. Compliance reduces the likelihood for major tool errors

Tests range from simple arithmetic operations to communication operations and verdict handling.

ATS checks syntax and semantics. ATS does not check the correctness of every single tool

message, but of the overall output.

TTCN-3 ToolTTCN-3

Conformance ATSMeaning?

pass



Challenges (1/2)

Various standard clarifications were necessary: 13 CRs for TTCN-3 core languages have been written. Unclarified test cases have been excluded from the delivery

("on-hold"). Tools sometimes handle the standard differently.

Tests need to be expressed as tool independent as possible.

Test Automation: No standard command line interface for batch run Tool-specific adapters No common test log format

TTCN3 semantics TTCN3 standard part 1 defines both static and dynamic

semantics No definition of well- and ill- formed programs

What to check at compile time is tool-dependent



Challenges (2/2)

Synchronize ATS with new TTCN3 revisions Time consuming manual activity As ATS grows the effort will grow as well To which core language standard version does the

conformance test suite relate to? How to maintain compatibility across different language

versions? Against which version should the conformance test suite be

developed in the future?

Which TTCN3 revision to support Should we certify newest TTCN3 revision only?



Conclusion

The conformance test suite is available now! ATS Metrics:

Total number of conformance tests: 1077 (340 new) 20 tests fixed 9 removed since deprecated or duplicates

Number of positive syntactic conformance tests: 138 Number of negative syntactic conformance tests: 82 Number of positive semantic conformance tests: 517 Number of negative semantic conformance tests: 340 Number of clauses (at least partially) covered: 134 (24 new)



30

TTCN-3 CONFORMANCE TEST SUITE STF 433

STATUS REPORT

Nikolay Pakulin

Requirements tracking case study



Case study in a nutshell

4 chapters: total 7,5 pages 12 Declaring messages, 13 Declaring timers, 23 Timer operations, 24 Verdict operations;

56 requirements elicited, 7 requirements refined into 34 test purposes; covered by 174 test cases

Average 7,5 requirements per page, 3 test cases per requirement



Requirements tracking case study

ISP RAS contributed to the project requirements management tool Requality Hierarchical grouping of requirements Requirements elicitation from DOC/HTML documents

Requirements are connected to certain locations in the documents

Requirements locations are highlighted Requirements refinement through test purposes

Test purposes define specific cases to be tested in order to treat corresponding requirement ‘covered’

Requirements coverage by test cases report Source document change management (beta)

Update requirements when the text of the standard is modified

Requality is an open-source plugin to eclipse Integrated with version control software (SVN)



Sample view: Requality in TTWorkbench



/*************************************************** ** @author STF 433 ** @version 0.0.1 ** @purpose 1:12, Ensure timer can be declared in components ** @verdict pass accept, ttcn3verdict:pass ***************************************************//* * #reqname /Requirements/12 Declaring timers/Timers can be declared in component type **/module Sem_12_toplevel_timer_001 { type component TComp{ timer t_timer:=10.0; } testcase TC_Sem_12_toplevel_timer_001() runs on TComp{ t_timer.start; if (t_timer.running){ setverdict(pass) } else { setverdict(fail) } } control{ execute(TC_Sem_12_toplevel_timer_001()) }}

Requirements Tracking (1/2)

Fully qualified name of the requirement



Requirements Tracking (1/2)

Requirements coverage report Integrated into

Eclipse Evaluates

coverage of requirements and test purposes



Suggested use case

Add requirements markup to TTCN3 standard Manual

Introduce references between requirements and TTCN3 test cases Manual

Track requirements changes between TTCN3 standard revisions Automated (beta)

Track required changes in TTCN3 ATS Manual, to be automated

TTCN3 standard updated

Changes in reqs.

reviewed

New reqs. added

ATS updated


Thank You!

Questions?


etsi update on ttcn-3 language evolution and tool conformance testing nikolay pakulin contributors:...

Documents