TSINGHUA SCIENCE AND TECHNOLOGY ISSN 1007-0214 09/49 pp51-56 Volume 12, Number S1, July 2007

Novel Software Automated Testing System Based on J2EE*

PEI Songwen (裴颂文)**, WU Baifeng (吴百锋), ZHU Kun (朱 琨), YU Qiang (余 强)

Department of Computing and Information Technology, Fudan University, Shanghai 200433, China

Abstract: Software automated testing is one of the critical research subjects in the field of computer applica-

tion. In this paper, a novel design of architecture called automated testing system (ATS) is proposed. Based

on techniques relating to J2EE including MVC design pattern, Struts framework, etc, ATS can support any

black-box testing business theoretically with relevant APIs programmed using Tcl script language before-

hand. Moreover, as the core of ATS is built in Java, it can work in different environments without being re-

complied. The efficiency of the new system is validated by plenty of applications in communication industry

and the results also show the effectiveness and flexibility of the approach.

Key words: automated testing system (ATS); J2EE; Tcl

Introduction

Software testing[1,2] is a critical phase within the soft-ware lifecycle, currently more and more companies are paying attention to it. Especially, the software auto-mated testing is becoming the most disputed subject in the software industry. Therefore, it is now a popular re-search problem in the computer application research area. The increased complexity of systems as well as short product release schedules makes the task of test-ing challenging. Therefore, how to improve the soft-ware qualities and how to advance the testing effi-ciency are difficult.

As a result, software testing is the biggest challeng-ing work during the software lifecycle, especially the software automated testing. Subsequently, test-driven development is advocated by engineers. At the same time, software capability maturity model[3] is put for-ward for controlling the process of software develop-ment and for guarantying the software quality, but it

imposes limited effects on software automated testing. To our knowledge, most of the present automated test-ing tools or products concern how to partition cases or how to implement a single target of testing without uniform testing system, not to mention the distributed and cross-platform testing functions.

However, ATS is a black-box testing system based on J2EE[4] theories such as MVC design pattern, Struts framework, Hibernate persistence layer, etc, which comprises two function modules: one is a front-end module developed in Java script language, the other is a back-end module — the core of ATS — developed in pure Java language. Thus, the two function modules abide by browser/server model naturally. Under the strong support of front-end module, testers without any prerequisite knowledge can master it in a short training time. Besides, the core of ATS can support cross-platform execution due to pure Java code, and it can support distributed execution due to the separation of job controller and execution server. Every testing busi-ness is supported theoretically by ATS due to the excel-lent inherent capability – self-defined APIs which can be developed by advanced testing developers quickly with Tcl[5] script language aiming at different special targets, and the APIs is transparent to common testers

﹡

﹡﹡

Received: 2007-02-01 Supported by the AMD’s University Cooperation Project To whom correspondence should be addressed. E-mail: 061021045@fudan.edu.cn; Tel: 86-21-61445936

Tsinghua Science and Technology, July 2007, 12(S1): 51-56

52

who work on the front-end system, what they need to do is just filling suitable data into blanks of web inter-face according to specific cases.

1 Related Work

The most popular methods about automation test in-clude record-playback, functional decomposition, data-driven[6], and keywords-driven[7]. Record-playback is applied in GUI test products usually, such as Winrun-ner1, etc. Functional decomposition method is an in-fant version of data-driven or keywords-driven method, which separates data from functions and uses data-files to provide both the input and the expected-results veri-fication. However, tester using this method must not only maintain the detail test plan with specific data, but also re-enter this data in the various required data-files. Data-driven method records data by a spreadsheet file such as Excel and the executions of test scripts depend-ing on various APIs and key data. Keywords-driven is an advanced data-driven method, which uses the actual test case document developed by the tester using a spreadsheet containing special keywords. In this method, the entire process is data-driven as well and the keywords control the processing. However, ATS has strong programming function supported by various fundamental and business’s APIs and flexible man-agement and control function due to combining data-driven and keywords-driven methods.

In recent years, there are many products and archi-tectures surged out. Blackburn, for instance, proposed a model-based test automation method called test automation framework (TAF)[8] which uses an inter-face-driven approach that combines requirements modeling to support automated test-case and test-driver generation. The model focuses on how test en-gineers can develop more reusable models by clarify-ing textual requirements as models in terms of compo-nent or system interfaces. However, due to un-canonical requirement documents and un-strict busi-ness control, it is very difficult to extend it to many big software companies, not to mention those middle-small companies. Meanwhile, the methods based on compo-nent test are proposed, such as CBSFG[9], etc. Unfortu-nately, they care little about the architecture of auto-mated testing framework but pay attention to how to generate test cases. iSoftTech2 test automation system and AXE3 automated testing framework are excellent

automation systems, but without distributed and cross-platform functions. The dominating commercial prod-ucts such as Winrunner and Robot4 are designed with-out much consideration about compatibility and port-ability. These traditional testing products care little about the operations on cross-platform, distributed control, and distributed execution capabilities.

2 ATS Architecture

ATS is developed according to Struts and MVC design pattern based on J2EE. Server end is built on the tom-cat server while client end is presented on Web pages programmed in JSP, and back-end business logic mod-ule is implemented by Java beans or even enterprise Java beans. The general ATS architecture is shown in Fig. 1[10-13].

Fig. 1 Architecture of ATS

The ATS is made up of user interface, request han-dler manager, job controller, normal request handlers, execution server, and suite execute layer. The user in-terface module provides front-end user with conven-ient operational interfaces to send the user’s operation requests to the request handler manager module. How-ever, the request handler manager module will not deal with these requests by itself, but just forward them to some specific handler or action threads.

In fact, there are two kinds of user request. One is related to control logic belong to test suite or test job, the other is miscellaneous request except from the first class. All the requests related to control logic are dealt with by job controller. To support distributed execution, job controller and execution server can run on different servers independently. A job controller can control sev-

PEI Songwen (裴颂文) et al：Novel software Automated Testing System Based on J2EE

53

eral execution servers synchronously. And execution server is designed by a finite states machine, which can run test suites queue in a test job. When all these test suites in a job queue is finished, the execution server will notify job controller to change the states of job. Of course, the suite execute layer is responsible for com-piling and executing suites or cases to implement a specific test task, which is an execution core of ATS. Meanwhile, it provides a unified testing framework with various APIs for programming test scripts at ease.

While LRMS is a library resource management sys-tem interfaced with ATS for promoting the manage-ment of test tasks and resources allocation. TMS is a testing management system interconnected with ATS for advancing the management of test results and test-ing version control. They are important auxiliary sys-tems supporting ATS.

2.1 Request handler manager

The front-end interfaces of ATS are Web pages. Tester can program test scripts or drag methods from the ba-sic library programmed in Tcl scripts to form a com-plete program segment. All these requests will be dis-patched to server. Generally, every request from front-end interface has its own handler function in the execu-tion server. If Web browser sends every request to the corresponding handler immediately, the Web browser must store all these relationships between request and handler function that will decrease the flexibility and extensibility of the whole system.

To solve this problem, ATS creates a request handler manager module to store these relationships between request and handler function. Figure 2 shows the task and the role of request handler manager in the ATS.

Fig. 2 Request handler manager in ATS

Thus, Web browser gets rid of this perplexity. It just needs to hand on requests to the request handler

manager which is responsible to dispatch them to cor-responding handler threads. Then, handler processes these quests through hibernate between handler and da-tabase, which returns results to Web browser finally. The request event sequence is shown in Fig. 3.

Fig. 3 Request event sequence diagram

When the request arrives, the request handler man-ager will retrieve information from the requests pool and construct an object of ActionForm bean with the information. Then, it will search an action from action mapping file for the request. If the action is found, the manager will forward it and get the corresponding da-tabase files. If not, it will dispatch the request to han-dler which is responsible for getting or updating corre-sponding database files.

The basic function of ATS is to build Web system based on struts framework and operating database through hibernate, which is our infant version of ATS called ATF. However, the ATF lacks modules to deal with the logic relationship between requests and han-dler events. We create job controller run as a finite states machine to solve this problem.

2.2 Job controller

In ATS, there are three kinds of test events—job, suite, and case. Job is an integrator of suite queues, which can be scheduled and allocated to an appointed execu-tion server. Suite is an implementing entity, which is assigned to suite execute layer by execution server. Case is an executable scripts designed in accordance with a certain business in Tcl script language.

Firstly, we overview job event in Fig. 4, then describe the finite states machine working in the job controller which is responsible for job control, such as job create, job copy, job schedule, job kill, and job query. The suite execution server is explained in the next segment.

Tsinghua Science and Technology, July 2007, 12(S1): 51-56

54

Fig. 4 Job event flow in ATS

Figure 4 shows a job controller maps n job executers, and a job executer maps n suite executer. Job controller, job executer, and suite executer can update or query their files storied in database, respectively, through dif-ferent interfaces. Obviously, job controller is responsi-ble for controlling the states of jobs and assigning jobs to different job executers for distributed execution. So，the job controller plays a key role in the whole working, which guarantees the entire system to run smoothly. Generally, different phases have different states. To control all these states well, we design a fi-nite states machine. Figure 5 shows the states transition of job controller’s finite states machine.

Fig. 5 Job state transition

As shown in Fig. 5, there are new state, suspended state, wait_start state, running state, closed state, and done state here. The contents contained in rectangle linked by broken line telling why job’s state transmits

from one to another. In order to bind jobs’ states with corresponding operations and to discriminate from the same or similar operations occurring in other states, the state pattern[14] is adopted to deal with these problems according to the theories of design pattern. We design a JobStateMachine class to drive the transition of job’s states. The class holds its instance all the time, which is designed in accordance with skeleton pattern. Job-StateMachine stores all states and other information of jobs into database. Besides, we design a basic class JobState, several subclasses derived from it to handle various jobs’ operations such as kill job, suspend job, etc. Another duty of job controller is to retrieve a cor-responding execution server according to the respec-tive configure file information of jobs. Specially, job controller and execution server can run on different servers simultaneously through RMI communication bridge, as a result, ATS can work in distributed mode naturally.

2.3 Execution server

As shown in Fig. 1, the execution server is an interme-diary of job controller and suite execute layer. Their re-lationships are shown in Fig. 6 which exposes the processes of execution server totally. Job controller launches a job to execute according to its state. Execution server receives execution job command and builds a suites queue for it, which will call suite execute layer to execute these suites in turn. When execution server receives the results returned

PEI Songwen (裴颂文) et al：Novel software Automated Testing System Based on J2EE

55

Fig. 6 Relationship among job controller, execution server, and suite execute layer

from suite execute layer, it will check whether the suite queue is null. If not, it will notice the next suite to exe-cute in the suite execute layer. Otherwise, the execu-tion server will inform job controller that all suites be-long to the job have been done.

Obviously, the suite execute layer is constituted by some executable scripts or commands to implement the task of a case. These scripts can be designed in Tcl language in a certain templates which can speed up the rate of testing development and guarantee the qualities of testing programs. The other function of the suite execute layer is that it provides APIs and external in-terfaces such as GUI API, Socket API to support a third tool -- Winrunner or QTP. Therefore, ATS has powerful flexibility and expansibility.

3 Experiments

As a matter of fact, we consider that all testing tasks can be classified into two classes: one is a testing about back-end program or procedure, the other is Web test or GUI test. For simplicity, the experiment about GUI test will be omitted here. So, we will only introduce an experiment about back-end program here.

Our experiment environment is listed as follows: the database is Oracle 9i.2.0, testing script is programmed in Tcl and executed on ActiveState ActiveTcl 8.5.0.0, Web server is tomcat 5.0. All of these are installed and running on Redhat operation system separately. The tested object is ShpptCDR which transforms a pure text file to another format file.

Our testing task is to check whether a certain tag in the transformed file cohering with what we expect ac-cording to transformation rules. The critical testing script about the transformation of 281th tag of source file is shown and explained below.

TestCase 1105432426903405902 0237243 {} { } { #Parameter section, this section is used by declaring and set-

ting global or local #parameters. } { # Setup section, commands in this section are to #prepare for

test, such as reserved the environment #variables. For example, S "" remoteoper GetRemoteFile "$CfgPath" "$Config File"; # The command is getting a remote configure file to local

host for the purpose of protecting configure #file from changed by testing.

. . . } { #Step sections, commands in this section are main executable

scripts. For example, M macro0; # Calls macro macro0 to pave the way for testing

S "" remoteoper RunRemoteCmd "$Command&" "$Prompt" ""; # Runs ShpttCDR on the remote host, not local. . . . S "" remoteoper RunRemoteCmd "$CtrlCCmd" "$Prompt" ""; # Executes Ctrl+C command to end the ShpttCDR program. . . . S "" toolkit CheckListValue "$ExpectTag281" "$Tag281" "1"

"EQUAL"; #checks the tag281 whether equal to what we expect. } { #Cleanup section, commands in this section is to recover test

environment. For example, M macro1; #Call macro macro1 to recover test environment to the situa-

tion that testing program have not been run. }

The script is not a pure Tcl script language, the in-structions like “S” and “M” will be transformed into pure Tcl scripts according to an instruction configure file. The result of this case running on single execution server is shown in Fig. 7.

Fig. 7 Testing result of ShpptCDR

The result shows that there is a case in testing the function suite – patch for centrex of ShpttCDR. The result log shows that this test is “PASS”, which dem-onstrates that the function bit tested tag281 is verified

Tsinghua Science and Technology, July 2007, 12(S1): 51-56

56

correct. And, the detailed log can be obtained from the step log label by clicking the left mouse key.

Meanwhile, we design another supplementary ex-periment that three jobs run on different execution server from execution servers 1 to 3, respectively, to test the function of ShpttCDR. The testing statistic re-sults are shown in Fig. 8.

Fig. 8 Experiment results statistics graph

As shown in the graph, there are 30 different cases corresponding to 30 different testing tags scattered in the transformed file. The 30 cases are arranged into a job with one suite running on three different servers by a distributed way. The statistical results show that, atexecution servers 1 and 2, the same results indicate that the suite run on servers 1 and 2 in a distributed way successfully. While the situation at the execution server 3, a case killed and another right case edited to error additionally on purpose, has been indicated in the graph correctly compared with the situations 1 and 2. In addition, by checking the detail log, all the cases have been executed in the purpose we expected. All these results prove that ATS can run in a distributed way, being controlled randomly and work perfectly.

4 Conclusions

Compared with traditional automated testing tools, ATS is a novel software automated testing system ap-plied to a communication company by plenty of suc-cessful test (such as GUI test with the help of Winrun-ner and back-end program test, etc.). It has been re-garded as an effective and novel testing system with distributed function, resource management function, and strong expansibility. Specially, we can design dif-ferent APIs to implement different testing business and

the scripts of regression test can execute on ATS with-out revising any command line but just revising global input variables. In a sense, the ATS is a general auto-mated testing framework.

However, the function of ATS should be improved in tracking of run-time instructions. And, the function of importing and exporting scripts between Web editor and database files should be created to gear to many testers who are not good at coding.

References

[1] Shan Jinhui, Jiang Ying, Sun Ping. Research progress in software testing. Acta Scientiarum Naturalium Universita-tis Pekinensis, 2005, 41(1): 134-145. (in Chinese)

[2] Graves Todd L, Harrold Mary Jean, Kim Jung-Min, et al. An empirical study of regression test selection techniques. ACM Transactions on Software Engineering and Method-ology (TOSEM), 2001, 10(2): 184-208.

[3] Paulk M, Weber C. The Capability Maturity Model Gride-line for Improving the Software Process. MA: Addison-Wesley, 1995.

[4] http://java.sun.com/javaee/, 2006. [5] Welch B B. Practical Programming in Tcl and Tk. New

York: Prentice Hall PTR, 1997. [6] Keith Zambelich. Totally data-driven automated testing,

http://www.sqa-test.com/w-paper1.html, 2006. [7] Feng Yucai, Tang Yan, Zhou Chun. Theory and implemen-

tation of keyword-driven test automation. Computer Ap-plications, 2004, 24(8): 140-142. (in Chinese)

[8] Blackburn M, Busser R, Nauman A. Interface-driven, model-based test automation. Crosstalk: The Journal of Defense Software Engineering, 2003: 27-30.

[9] Smai Beydeda, Volker Gruhn. An integrated testing tec-nique for component-based software. In: Proceedings of the ACS/IEEE Int. Conf. on Computer Systems and Appli-cations (AICCSA'01). Aiccsa, 2001: 328-334.

[10] http://www.mercury.com. [11] http://www.isofttech.com/. [12] http://www.odin.co.uk/. [13] http://www-306.ibm.com/software/rational/. [14] Erich G, Richard H, Ralph J, John V. Design Patterns Ele-

ments of Reusable Object-Oriented Software. MA: Addi-son-Wesley, 1995.