software testing a craftmans approach

|��

A-PDF Merger DEMO : Purchase from www.A-PDF.com to remove the watermark

http://www.a-pdf.com

�

ë��ë��

�� `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

£

ë��ë��

ºº `�� `�� ºº `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

Ñ

��ë��ë��

�� &(�&(�))*��*��+*��+*��,��,��)�)�,��*��,��,��*��,��,��,��

�� `��#�� `��#��

�� #�� #�� #�� #�� #�� #��

�� -�-�� (.�� ,+*�� -�-�� (.�� ,+*��

r

��ë��ë��

�� `��/�� #��#�� `��/�� #��#�� (��+*�0�� #� ��(��1+*��(��+*�0�� #� ��(��1+*��#��(��+*�0�� 2� ��#��(��+*�0�� 2� ��(��(��33+*�� (��++*�� (��+

�� 4�� 5�� /�4�� 5�� /�� 2��(��+�� -�� 2��(��+�� -

�� #�� #�� #�� #�� #�� #��*�� #�� #��*�� #�� #��#�� #��

]

��ë��ë��

&(�)*��+

X

ë��ë��

ºº `�� `�� ºº `��`��

ºº ��`��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

�

3��ë��3��ë��

�� `�� #��`�� #�� &��#��&��#��6�Ñ��1�)� � ��6�Ñ��1�)� � ��

�� `�� %��*�� (�� +��`�� %��*�� (�� +�� #�� #�� #�� *��#��#�� *��#��

�� -�-�!� "� �� *��-�-�!� "� �� *��

�� #�� #�� 4�� 5�� 4�� 5��

�� -�-� ��-�-� ��

�� `��#��`��#�� "�� "�� 33��

�� #��(�� #�� #��+��#��(�� #�� #��+�� -�-� 7!�� 8�� -�-� 7!�� 8��

ù

ë��ë��

ºº `�� `�� ºº `��`��

�� `��`��ºº �� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

)9

a��ë��a��ë��

�� `��#��2��%��2�2��`��#��2��%��2�2�� #��#��m��m�� #�� #�� m��m� $�� $��

�� -�-�!� "� � � ��$�� (� �-�-�!� "� � � ��$�� (� �� &��*�� &��*�� *��*��+�� *��*��+

�� -�-�#�� $�� *�-�-�#�� $�� *�� *��*�� -�� *��*�� -

))

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

ºº ��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

)�

¦��¦��

�� #�� #��

�� 7�� #��#��7�� #��#��#��#��*��2�� 2��#��#��*��2�� 2�� 2� ��#�� 2� ��#�� (��1+��#�� (��1+��#�� (�� (��33++

�� &�� &��

�� -�-�� #� ��-�-�� #� ��

�� &�� &��

)£

¦��¦��

)Ñ

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

�� ºº ��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

)r

Ô��Ô��

�� /�� #�� /�� #�� #�� #��

�� / ��/�� &��#�� .��*��1*��*��&��#�� .��*��1*��*��33*��:*��:

�� %� �� %� ��

�� 8�� #��8�� #��

�� &��#��6�r&��#��6�r�� (�� Ñ�1)� � �� (�� Ñ�1)� � ��#��+��#��+

�� &��2�� &��2��

��

�� 2�� 2��

)]

Ô��Ô��

)X

Ô��Ô��

�

�

�

�

�)

�� (�� 2�

#��+

)�

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

�� ºº ��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

)ù

��

�� 2�� 2��

�� 8�� #*�� *��8�� #*�� *�� ;��%��2��*�� ;��%��2��*��2� ��*�4��2� ��*�4�� 5 � ��5

�� 7 �� 7 �� #�� 0� �#*�� #�� 0� �#*�� 2�� 2�� #��#�� #��#��

�9

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

�� ºº ��

ºº �� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

�)

��

�� '� ��

) )99 )99 ) 7��

� )99 )99 � 7��

£ )99 )99 )99 �$��

Ñ )99 )99 )ùù 7��

r )99 )99 �99 !� ��

] )99 ) )99 7��

X )99 � )99 7��

� )99 )ùù )99 7��

ù )99 �99 )99 !� ��

)9 ) )99 )99 7��

)) � )99 )99 7��

)� )ùù )99 )99 7��

)£ �99 )99 )99 !� ��

`��

��<�)��1�<��<�)99��3 <�)ùù��<��99

��

��

� � � �

� �

� � � ��

= � ��

> ?? � ��

@ � � ��

A � � ��

B � � 7� ��

C � � ��

? � ?? � ��

� � � ��

� ��

� � � ��

= 7� ��

> ?? � ��

@ � � ��

A ?? � ��

B ?? � � ��

C ?? � ��

? ?? ?? 7� ��

� ?? � � ��

� � � ��

��

�= � � ��

�> � ?? � ��

�@ � � 7� ��

�A � � ��

�B � � 7� ��

�C � � ��

�? � ?? � ��

= � � � ��

� � � �

= � � 7� ��

=� � � � � �

== � � � ��

=> � � ?? � ��

=@ � � � � ��

=A � � ��

=B � � � ��

=C � 7� ��

=? � ?? � ��

> � � � ��

> � ?? � ��

>� � ?? � � ��

>= � ?? � ��

>> � ?? ?? 7� ��

>@ � ?? � �

>A � � � ��

>B � � � � ��

>C � � � ��

>? � � ?? �

@ � � � 7� ��

@ � ��

@� � � ��

@= 7� ��

@> ?? � ��

@@ � � ��

@A � � ��

@B � � � ��

@C � 7� ��

@? � ?? � ��

A � � � ��

� �� (A � �� @+

�£

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

�� ºº ��

�� ºº �� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

�Ñ

�� !�� !��

`��

�� '� ��

) ] )r )�)� �D��)]*�)�)�

� ] )r )�)£ �D��)]*�)�)£

£ ] )r )ù)� �D��)]*�)ù)�

Ñ ] )r �9)) �D��)]*��9))

r ] )r �9)� �D��)]*��9)�

] ] ) )ù)� �D��*�)ù)�

X ] � )ù)� �D��£*�)ù)�

� ] £9 )ù)� �D��)*�)ù)�

ù ] £) )ù)� ��

)9 ) )r )ù)� �D��)]*�)ù)�

)) � )r )ù)� �&��)]*�)ù)�

)� )) )r )ù)� �!�#��)]*�)ù)�

)£ )� )r )ù)� �"��)]*�)ù)�

�� <�)��1�<��<�]��3 <�))��<�)�

��<�)��1�<��<�)r��3 <�£9��<�£)

��<�)�)��1�<�)�)£��<�)ù)��3 <��9))��<��9)�

�r

�� (]9��)�r+

�� !�� !��

��

) ) ) )C)� �D� �� *�)C)�

� ) ) )C)= �D� �� *�)C)=

= ) ) )?)� �D� �� *�)?)�

> ) ) �9)) �D� �� *��9))

r ) ) �9)� �D� �� *��9)�

] ) � )C)� �D� �� =*�)C)�

B ) � )C)= �D� �� =*�)C)=

C ) � )?)� �D� �� =*�)?)�

? ) � �9)) �D� �� =*��9))

)9 ) � �9)� �D� �� =*��9)�

)) ) )r )C)� �D� �� )]*�)C)�

)� ) )r )C)= �D� �� )]*�)C)=

)= ) )r )?)� �D� �� )]*�)?)�

)> ) )r �9)) �D� �� )]*��9))

)r ) )r �9)� �D� �� )]*��9)�

)] ) =9 )C)� �D� �� =)*�)C)�

)B ) =9 )C)= �D� �� =)*�)C)=

)C ) =9 )?)� �D� �� =)*�)?)�

)? ) =9 �9)) �D� �� =)*��9))

�9 ) =9 �9)� �D� �� =)*��9)�

�) ) =) )C)� � � �� )*�)C)�

�� ) =) )C)= � � �� )*�)C)=

�= ) =) )?)� � � �� )*�)?)�

�> ) =) �9)) � � �� )*��9))

�r ) =) �9)� � � �� )*��9)�

�] � ) )C)� � � �� *�)C)�

�B � ) )C)= � � �� *�)C)=

�C � ) )?)� � � �� *�)?)�

�? � ) �9)) � � �� *��9))

=9 � ) �9)� � � �� *��9)�

��

=) � � )C)� � � �� =*�)C)�

=� � � )C)= � � �� =*�)C)=

== � � )?)� � � �� =*�)?)�

=> � � �9)) � � �� =*��9))

=r � � �9)� � � �� =*��9)�

=] � )r )C)� � � �� )]*�)C)�

=B � )r )C)= � � �� )]*�)C)=

=C � )r )?)� � � �� )]*�)?)�

=? � )r �9)) � � �� )]*��9))

>9 � )r �9)� � � �� )]*��9)�

>) � =9 )C)� ��

>� � =9 )C)= ��

>= � =9 )?)� ��

>> � =9 �9)) ��

>r � =9 �9)� ��

>] � =) )C)� ��

>B � =) )C)= ��

>C � =) )?)� ��

>? � =) �9)) ��

r9 � =) �9)� ��

r) ] ) )C)� �D ��*�)C)�

r� ] ) )C)= �D ��*�)C)=

r= ] ) )?)� �D ��*�)?)�

r> ] ) �9)) �D ��*��9))

rr ] ) �9)� �D ��*��9)�

r] ] � )C)� �D �=*�)C)�

rB ] � )C)= �D �=*�)C)=

rC ] � )?)� �D �=*�)?)�

r? ] � �9)) �D �=*��9))

]9 ] � �9)� �D �=*��9)�

�]

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

�� ºº ��

�� !� "� � �� !� "� � ��ºº ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

�X

��

�� )£�`�� )£�`��

�� )�r�� )�r��

�� `��#�� *��`��#�� *�� E)999��E)�99�� E)999��E)�99

�� 9F�� 9F��

�� 2�� #�� 2�� #�� #��/�E)99*�E)999*�� #��/�E)99*�E)999*�E)�99*��EX�99E)�99*��EX�99

��

��

�� %� � �%� ��

� � �

� � �@ �-@ � � � �

= � = = � � � �

> � >@ >-@ � � � �

@ @ @ @ @ @ � �

A ? ?B@ ?B-@ � �� 3

B ? ?B ?B � �� 3

� ? ?@@ ?@-@ � �� 3

? � ��

�@ =-B@ � ��

= >-@ � ��

� >@ A-B@ � ��

= > > > > A � �

> � �� B �BB@ ��A-�@ � �� 3

�@ �� B �� BB ��@-@ � �� 3

�A �B �� B@@ ��=-�@ � �� 3

�B ��

�� ? ��@ ��@ ��

�? �� ? �� =� ��A ��

�� ? �� >@ ��? ��

�� >� >� >� >��

�� B� �� ? BBB@ �>�@ �� 3

�= B� B? ?� BBB� �>�> �� 3

�> A? �� ?� BB@@ �>�� 3

�@ B� �� ?� B�� >��

'� � �`��

�ù

��

�� ù�� E)999�� ù�� E)999��

�� 7��2� 2�%� �� #��2�� 7��2� 2�%� �� #��2�� #��0�� 2��0�� #� ��#��0�� 2��0�� #� ��

�� &��#�� &��#��

�� %� � ��%� ��

) )9 )) ù )99r )99-Xr �� 1

� )� )X )ù )Xùr �)ù-�r �� 3

£ )� )ù )X )�9r ��) �� 1

'� ��

£9

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� ºº �$��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

£)

��

�� 8� �#� ��8� �#� �� G�#�� G�#�� #��#��

�� $��#�� *�2�� $��#�� *�2�� 0�� 2�� 0�� 2�� (�� *�� (�� *��33��+��+

�� $��#��$��#��

�� 4 �� 5�4 �� 5�6�4 ��#�� 56�4 ��#�� 5�� %�� $��#�� %�� $��#��

�� 4��5� ��%�� *�� %�� 4��5� ��%�� *�� %�� 2�� 2��

£�

��

�� / �� /

�� &�� #��*��*��&�� #��*��*��

�� 7�� *�`*��7�� *�`*��

�� <��)��£��<��)��£

�� `�<�`)��`��`£��`Ñ`�<�`)��`��`£��`Ñ

�� <��)��<��)��

�� )��)*��£��`£*��)��)*��£��`£*��

X

Ú

X

X X X

X

Ú Ú

££

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� ºº �$��#��$��#��

ºº ��%��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

£Ñ

Ô�"��Ô�"��

�� #��#��$��#�� $��#��

�� !��2�%�$��#�� !��2�%�$��#�� <�� <�� 2� �� 2� ��

��

� ��

� � ��

� = �= �= ��

� > �� > ��

��<��=`�<�`��`��`=��`>

��<��*��=��`=*��

X X

X X X

X

Ú Ú Ú

£r

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� ºº �$��#��$��#��

�� %��$��#�� %��$��#�� ºº � ��$��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

£]

��

�� `�� `��

�� `��<�£��Ñ��<��Ñ��`��<�£��Ñ��<��Ñ��

�� $��#�� $��#��

��

��

��£ ��

��Ñ ��

��@ �� £ ��

��A �� £ ��

��B �� Ñ ��

��C �� Ñ ��

��? ��

��

��

��£ �� £ ��

��Ñ �� £ ��

��@ �� Ñ ��

��A �� Ñ ��

��B �£ ��

��C �£ ��

��? �£ ��

�� £ �� £ �£ ��

�� £ �£ ��

��£ �£ �Ñ ��

��Ñ �£ �Ñ ��

£X

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� ºº �$��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� ºº �� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

£�

��

�� "��$��#�� #�� "��$��#�� #��

�� /�)�,��%�,�X9*�)�,�� %�,��9*�� /�)�,��%�,�X9*�)�,�� %�,��9*��)�,��,�ù9)�,��,�ù9

�� 7�#�� /��%�.�)*��%�:�X9*�� %�.�)*��7�#�� /��%�.�)*��%�:�X9*�� %�.�)*�� %�:��9*��.�)*��:�ù9� ��%�:��9*��.�)*��:�ù9

�� &��#�� *��#��#��&��#�� *��#��#��(�%�2�%�$��#�� +��(�%�2�%�$��#�� +

�� &��#�� *�� 2��#��&��#�� *�� 2��#��#��#�� #��2��#��#�� #��2��#��(��+#��(��+

£ù

��

�� / ��/

�� *� �� *� �� 2�� #�� 2�� #�� 6�� 6��

�� #�� #��

Ñ9

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

ºº �$��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

Ñ)

��

��

�� '� �� /�!� ��*��*�7��*��$�� '� �� /�!� ��*��*�7��*��$��

�� (��+�$��#�� (��+�$��#�� )�<�H.�*��*��:�/� �� 2� ��*��*��$�� I�)�<�H.�*��*��:�/� �� 2� ��*��*��$�� I

�� <�H.�*��*��:�/� �� 2� ��*��*��I��<�H.�*��*��:�/� �� 2� ��*��*��I

�� £�<�H.�*��*��:�/� �� 2� ��*��*��I�£�<�H.�*��*��:�/� �� 2� ��*��*��I

�� Ñ�<�H.�*��*��:�/��*��*�� I�Ñ�<�H.�*��*��:�/��*��*�� I

�� '� ��

'�) @ @ @ �$��

'�� £ 7��

'�£ £ Ñ @ ��

'�Ñ Ñ ) � !� ��

Ñ�

��

��

�� 7�� (��+�$��#��7�� (��+�$��#��

")�<�H.�*��*��:�/��<��<��I"��<�H.�*��*��:�/��<��*��J��I"£�<�H.�*��*��:�/��<��*��J��I"Ñ�<�H.�*��*��:�/��<��*��J��I

"r�<�H.�*��*��:�/��J��*��J��*��J��I;;�"]�<�H.�*��*��:�/��K��1��I"]��<�H.�*��*��:�/��<��1��I"]��<�H.�*��*��:�/��:��1��I;;�"X�<�H.�*��*��:�/��K��1��I"X��<�H.�*��*��:�/��<��1��I"X��<�H.�*��*��:�/��:��1��I;;�"��<�H.�*��*��:�/��K��1��I"��<�H.�*��*��:�/��<��1��I"��<�H.�*��*��:�/��:��1��I

Ñ£

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� ºº �$��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

ÑÑ

��

� �!��|�� !��|��

�� 7�� #��7�� #�� )�,�� ,�)�)�,�� ,�)�� )�,��,�£))�,��,�£)�� )�)��,��,��9)�)�)��,��,��9)�

�� $��#��$��#��

�� 8)�<�H�� /�)�,�� ,�)��I8)�<�H�� /�)�,�� ,�)��I�� ")�<�H��/�)�,��,�£)�I")�<�H��/�)�,��,�£)�I�� L)�<�H��/�)�)��,��,��9)��IL)�<�H��/�)�)��,��,��9)��I

�� 7�#��$��#��7�#��$��#�� 8��<�H�� /�� .�)�I8��<�H�� /�� .�)�I�� 8£�<�H�� /�� :�)��I8£�<�H�� /�� :�)��I�� "��<�H��/��.�)�I"��<�H��/��.�)�I�� "£�<�H��/��:£)�I"£�<�H��/��:£)�I�� L��<�H��/��.�)�)��IL��<�H��/��.�)�)��I�� L£�<�H��/��:��9)��IL£�<�H��/��:��9)��I

��7" 8�� "�� L�� '� �� ) ] )r )ù)� ];)];)ù)�

�� 3) )r )ù)� ��#��

��£ )£ )r )ù)� ��#��

��Ñ ] 3) )ù)� ��#��

��r ] £� )ù)� ��#�� ] ] )r )�)) ��#��

��X ] )r �9)£ ��#��

Ñr

��

� �!��|�� !��|��

�� 4 �� 5� �� 4 �� 5� �� #��;��#��#�� #��;��#��#

�� "�� /�� M"�� /�� M

8)�<�H�� /�� £9��I8��<�H�� /�� £)��I8£�<�H�� /�� &��I

")�<�H��/�)�,��,��I"��<�H��/��<��ù�I"£�<�H��/��<�£9�I"Ñ�<�H��/��<�£)�I

L)�<�H��/��<�)ù99�IL��<�H��/�)�)��,��,��9)��!"�(��J�)ù99+

�!"�(��Ñ�<�9+ IL£�<�H��/�)�)��,��,��9)��!"�(��Ñ�J�9+�I

!� �� $��#��NNN!� �� $��#��NNN

Ñ]

��

� �!��|�� !��|��

��7" 8�� "�� L�� '� ��

��) ] )Ñ )ù99 ];)r;)ù99�� X �ù )ù)� X;£9;)ù)�

��£ � £9 )ù)£ ��#��

��Ñ ] £) )ù99 ��#��

��%�$��#��

��7" 8�� "�� L�� '� ��

��) ] )Ñ )ù99 ];)r;)ù99

�� ] )Ñ )ù)� ];)r;)ù)�

��£ ] )Ñ )ù)£ ];)r;)ù)£

��Ñ ] �ù )ù99 ];£9;)ù99

��r ] �ù )ù)� ];£9;)ù)�

��] ] �ù )ù)£ ];£9;)ù)£

��X ] £9 )ù99 9X;9);)ù99

��C ] £9 )ù)� 9X;9);)ù)�

��ù ] £9 )ù)£ 9X;9);)ù)£

��)9 ] £) )ù99 � '

��)) ] £) )ù)� � '

��)� ] £) )ù)£ � '

��)£ X )Ñ )ù99 X;)r;)ù99

��)Ñ X )Ñ )ù)� X;)r;)ù)�

��)r X )Ñ )ù)£ X;)r;)ù)£

��)] X �ù )ù99 X;£9;)ù99

��)X X �ù )ù)� X;£9;)ù)�

��)C X �ù )ù)£ X;£9;)ù)£

��7" 8�� "�� L�� '� ��

��)ù X £9 )ù99 X;£);)ù99

��9 X £9 )ù)� X;£);)ù)�

��) X £9 )ù)£ X;£);)ù)£

�� X £) )ù99 9C;9);)ù99

��£ X £) )ù)� 9C;9);)ù)�

��Ñ X £) )ù)£ 9C;9);)ù)£

��r � )Ñ )ù99 �;)r;)ù99

��] � )Ñ )ù)� �;)r;)ù)�

��X � )Ñ )ù)£ �;)r;)ù)£

��C � �ù )ù99 � '

��ù � �ù )ù)� 9£;9);)ù)�

��£9 � �ù )ù)£ � '

��£) � £9 )ù99 � '

��£� � £9 )ù)� � '

��££ � £9 )ù)£ � '

��£Ñ � £) )ù99 � '

��£r � £) )ù)� � '

��£] � £) )ù)£ � '

� ��$��#��

ÑX

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� ºº �$��#�� $��#�� '��#� ��'��#� ��

Ñ�

��

��

�� 7�� "��$��#��7�� "��$��#�� %��%

�� )�<�H��%/�)�,��%�,�X9�I�)�<�H��%/�)�,��%�,�X9�I�� <�H��%/��%�.�)�I��<�H��%/��%�.�)�I�� £�<�H��%/��%�:�X9�I�£�<�H��%/��%�:�X9�I

�� %� ��%�� )�<�H�� %/�)�,�� %�,��9�I�)�<�H�� %/�)�,�� %�,��9�I�� <�H�� %/�� %�.�)�I��<�H�� %/�� %�.�)�I�� £�<�H�� %/�� %�:��9�I�£�<�H�� %/�� %�:��9�I

�� `��`�� `)�<�H��/�)�,��,�ù9�I`)�<�H��/�)�,��,�ù9�I�� `��<�H��/��.�)�I`��<�H��/��.�)�I�� `£�<�H��/��:�ù9�I`£�<�H��/��:�ù9�I

Ñù

��

��

� ��7�� "��$��#��

�� %� � ��%� ��

�� =@ Ñ� Ñ@ =ù�� AÑ�

�� =@ Ñ� � � �

��= =@ Ñ� ù� � �

��Ñ =@ � Ñ@ � �

��@ =@ � � � �

��A =@ � ù� � �

��B =@ C� Ñ@ � �

��C =@ C� � � �

��ù =@ C� ù� � �

�� Ñ� Ñ@ � �

�� Ñ� � � �

�� Ñ� ù� � �

��= � � Ñ@ � �

��Ñ � � � � �

��@ � � ù� � �

��A � C� Ñ@ � �

��B � C� � � �

��C � C� ù� � �

��ù B� Ñ� Ñ@ � �

�� B� Ñ� � � �

�� B� Ñ� ù� � �

�� B� � Ñ@ � �

��= B� � � � �

��Ñ B� � ù� � �

��@ B� C� Ñ@ � �

��A B� C� � � �

��B B� C� ù� � �

�� %� � ��%� ��

�� =@ Ñ� Ñ@ =ù�� AÑ�

��

��= B� C� ù� � �

��%�7�� "��$��#��

r9

��

��

�� <�Ñr��%��1�£9�� %��1��r��<�Ñr��%��1�£9�� %��1��r��

�� )�<�H�.��%*�� %*��:�/��,�)999�I�)�<�H�.��%*�� %*��:�/��,�)999�I

�� <�H�.��%*�� %*��:�/�)999�.��,��<�H�.��%*�� %*��:�/�)999�.��,�)�99�I)�99�I

�� £�<�H�.��%*�� %*��:�/��:�)�99�I�£�<�H�.��%*�� %*��:�/��:�)�99�I

'� �� $��#��

�� %� � ��%� ��

'�) r r r r99 r9

'�� )r )r )r )r99 )Br

'�£ �r �r �r �r99 £A9

r)

ë��ë��

ºº `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

ºº �$��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� ºº ��'��#� ��'��#� ��

r�

3��#��3��#��

)-)- �� $��#�� $��#�� 2�%�$��#�� 2�%�$��#�� *�2�� *�� *�2�� *�� $��#�� $��#��

�-�- �� %�� %�� 2�� 2��

£-£- 7�� *�2�� 7�� *�2�� $��#�� #�� $��#�� #��(-�-��+��(-�-��+

Ñ-Ñ- �$��#�� 2�� $��#�� 2�� #��-�� 2�� #��-�� 2�� 33��33�� #��#�� #��#��

r£

3��#��3��#��

r-r- �$��#�� $��#�� 2� ��#�� -�(��4��5��2� ��#�� -�(��4��5� �� $��#��+� �� $��#��+�(-�-�!� "� �� +(-�-�!� "� �� +

]-]- �$��#�� 2�� $��#�� 2�� -�7��*� �� -�7��*� �� $��#��*�� !� "� �� $��#��*�� !� "� ��

X-X- � ��$��#�� %�� $��#�� %�� #�� 2�� #�� 2�� %�-�7�� *� �� %�-�7�� *� ��2�� 4��5� � ��*�� 2�� 4��5� � ��*�� !� "� �� !� "� ��

rÑ

3��#��3��#��

�-�- �#�� 4 ��#�� 4 �� 5�$��#�� #��*�� 5�$��#�� #��*��2��2�� !� "� ��-�7��2��2�� !� "� ��-�7�� *� ��4��#��5�� *� ��4��#��5��4�� 5�$��#�� -��4�� 5�$��#�� -�� *� �� *� ��

rr

ë��ë��

�� `�� `�� `��`��

�� `��`�� `�� `��

��

�� !� "� � �� !� "� � ��

�� `��#�� `��#��

�� $��#�� $��#�� $��#��$��#��

�� %��$��#�� %��$��#�� $��#�� $��#�� $��#�� $��#��

�� $��#�� $��#�� $��#�� !� "� �&�� $��#�� !� "� �&�� $��#�� $��#�� '��#� ��'��#� ��

r]

¦��¦��

�� 2�� O�� 2�� O�� *� ��-�D��*�� *� ��-�D��*�� (�� r��]+ ��(�� r��]+

Software Testing: A Craftsman’s Approach, 3rd Edition Equivalence Class Testing

Chapter 6

Equivalence Class Testing


Equivalence Class Testing

Domain Range

F

Equivalence class testing uses information about the functional mapping itself to identify test cases


Equivalence Relations

• Given a relation R defined on some set S, R is anequivalence relation if (and only if), for all, x, y, and zelements of S:– R is reflexive, i.e., xRx– R is symmetric, i.e., if xRy, then yRx– R is transitive, i.e., if xRy and yRz, then xRz

• An equivalence relation, R, induces a partition on the set S,where a partition is a set of subsets of S such that:– The intersection of any two subsets is empty, and– The union of all the subsets is the original set S

• Note that the intersection property assures no redundancy,and the union property assures no gaps.


Equivalence Partitioning

Domain Range

F Define relation R as follows: for x, y in domain, xRy iff F(x ) = F(y). Facts: 1. R is an equivalence relation. 2. An equivalence relation induces a partition on a set. 3. Works best when F is many-to-one 4. (pre-image set)

Domain Range

Test cases are formed by selecting one value from each equivalence class. - reduces redundancy - identifying the classes may be hard


Forms of Equivalence Class Testing

• Normal: classes of valid values of inputs• Robust: classes of valid and invalid values of

inputs• Weak: (single fault assumption) one from each

class• Strong: (multiple fault assumption) one from

each class in Cartesian Product• We compare these for a function of two

variables, F(x1, x2)• Extension to problems with 3 or more variables

is “obvious”.


Weak Normal Equivalence Class Testing

• Identify equivalence classes of valid values.• Test cases have all valid values.• Detects faults due to calculations with valid values

of a single variable.• OK for regression testing.


Equivalence classes (of valid values):{a <= x1 < b}, {b <= x1 < c}, {c <= x1 <= d}{e <= x2 < f}, {f <= x2 <= g}

Weak Normal Equivalence Class Test Cases

a b c d

x2

x1

e

f

g


Weak Robust Equivalence Class Testing

• Identify equivalence classes of valid and invalidvalues.

• Test cases have all valid values except one invalidvalue.

• Detects faults due to calculations with valid valuesof a single variable.

• Detects faults due to invalid values of a singlevariable.

• OK for regression testing.


Equivalence classes (of valid and invalid values):{a <= x1 < b}, {b <= x1 < c}, {c <= x1 <= d}, {e <= x2 < f}, {f <= x2 <= g}Invalid classes: {x1 < a}, {x1 > d}, {x2 < e}, {x2 > g}

Weak Robust Equivalence Class Test Cases

a b c d

x2

x1

e

f

g


a b c d

x2

x1e

f

g

a b c d

x2

x1e

f

g

Is thispreferable

to this? Why?


Strong Normal Equivalence Class Testing

• Identify equivalence classes of valid values.• Test cases from Cartesian Product of valid values.• Detects faults due to interactions with valid

values of any number of variables.• OK for regression testing, better for progression

testing.


Strong Normal Equivalence Class Test Cases

a b c d

x2

x1

e

f

g

Equivalence classes (of valid values):

{a <= x1 < b}, {b <= x1 < c}, {c <= x1 <= d}

{e <= x2 < f}, {f <= x2 <= g}


Strong Robust Equivalence Class Testing

• Identify equivalence classes of valid and invalid values.• Test cases from Cartesian Product of all classes.• Detects faults due to interactions with any values of any

number of variables.• OK for regression testing, better for progression testing.

(Most rigorous form of Equivalence Class testing, BUT,Jorgensen’s First Law of Software Engineering applies.)

• Jorgensen’s First Law of Software Engineering:– The product of two big numbers is a really big number.– (scaling up can be problematic)


Strong Robust Equivalence Class Test Cases

a b c d

x2

x1

e

f

g

Equivalence classes (of valid and invalid values):

{a <= x1 < b}, {b <= x1 < c}, {c <= x1 <= d}, {e <= x2 < f}, {f <= x2 <= g}

Invalid classes: {x1 < a}, {x1 > d}, {x2 < e}, {x2 > g}


Selecting an Equivalence RelationThere is no such thing as THE equivalence relation.

If x and y are days, some possibilities for Nextdate are:

•x R y iff x and y are mapped onto the same year•x R y iff x and y are mapped onto the same month•x R y iff x and y are mapped onto the same date•x R y iff x(day) and y(day) are “treated the same”•x R y iff x(month) and y(month) are “treated the same”•x R y iff x(year) and y(year) are “treated the same”

Best practice is to select an equivalence relation thatreflects the behavior being tested.


NextDate Equivalence Classes• Month:

– M1 = { month : month has 30 days}– M2 = { month : month has 31 days}– M3 = { month : month is February}

• Day– D1 = {day : 1 <= day <= 28}– D2 = {day : day = 29 }– D3 = {day : day = 30 }– D4 = {day : day = 31 }

• Year (are these disjoint?)– Y1 = {year : year = 2000}– Y2 = {year : 1812 <= year <= 2012 AND (year ≠ 0 Mod 100)

and (year = 0 Mod 4)– Y3 = {year : (1812 <= year <= 2012 AND (year ≠ 0 Mod 4)


Not Quite Right

• A better set of equivalence classes for year is– Y1 = {century years divisible by 400} i.e., century leap

years– Y2 = {century years not divisible by 400} i.e., century

common years– Y3 = {non-century years divisible by 4} i.e., ordinary

leap years– Y4 = {non-century years not divisible by 4} i.e., ordinary

common years• All years must be in range: 1812 <= year <= 2012• Note that these equivalence classes are disjoint.


Weak Normal Equivalence Class Test Cases

Select test cases so that one element from each input domain equivalence class is used as a test input value.

April 1 2000 Jan. 29 1900 Feb. 30 1812 April 31 1901

WN-1 WN-2 WN-3 WN-4

Test Case

Input Domain Equiv. Classes

Input Values Expected Outputs

M1, D1, Y1 M2, D2, Y2 M3, D3, Y3 M1, D4, Y4

April 2 2000 Jan. 30 1900 impossible impossible

Notice that all forms of equivalence class testing presumethat the variables in the input domain are independent;logical dependencies are unrecognized.


• With 4 day classes, 3 month classes, and 4 year classes,the Cartesian Product will have 48 equivalence class testcases. (Jorgensen’s First Law of Software Engineeringstrikes again!)

• Note some judgment is required. Would it be better tohave 5 day classes, 4 month classes and only 2 yearclasses? (40 test cases)

• Questions such as this can be resolved by consideringRisk.

Strong Normal Equivalence Class Test Cases


Revised NextDate Domain Equivalence Classes

The Cartesian Product of these contains 40 elements.

•Month:–M1 = { month : month has 30 days}–M2 = { month : month has 31 days except December}–M3 = { month : month is February}–M4 = {month : month is December}

•Day–D1 = {day : 1 <= day <= 27}–D2 = {day : day = 28 }–D3 = {day : day = 29 }–D4 = {day : day = 30 }–D5 = {day : day = 31 }

•Year (are these disjoint?)–Y1 = {year : year is a leap year}–Y2 = {year : year is a common year}


When to Use Equivalence Class Testing

• Variables represent logical (rather than physical)quantities.

• Variables “support” useful equivalence classes.• Try to define equivalence classes for

– The Triangle Problem• 0 < sideA < 200• 0 < sideB < 200• 0 < sideC < 200

– The Commission Problem (exercise)


Another Equivalence Class Strategy

• “Work backwards” from output classes.

• For the Triangle Problem, could have– {x, y, z such that they form an Equilateral triangle}– {x, y, z such that they form an Isosceles triangle with x = y}– {x, y, z such that they form an Isosceles triangle with x = z}– {x, y, z such that they form an Isosceles triangle with y = z}– {x, y, z such that they form a Scalene triangle}

• How many equivalence classes will be needed forx,y,z Not a triangle?


In-Class Exercise

• Apply the “working backwards” approach todevelop equivalence classes for theCommission Problem.

• Hint: use boundaries in the output space.


Assumption Matrix

Robust WorstCase BoundaryValueStrong RobustEquiv. Class

Worst CaseBoundary Value

Strong NormalEquiv. Class

Multiple fault

RobustBoundary Value

Weak RobustEquiv. Class

Boundary Value

Weak NormalEquiv. Class

Single fault

Valid andInvalid Values

Valid Values

Software Testing: A Craftsman’s Approach, 3rd Edition Decision Table Based Testing

Chapter 7

Decision Table Based Testing


Decision Table Based Testing

• Originally known as Cause and Effect Graphing– Done with a graphical technique that expressed

AND-OR-NOT logic.– Causes and Effects were graphed like circuit

components– Inputs to a circuit “caused” outputs (effects)

• Equivalent to forming a decision table in which:– inputs are conditions– outputs are actions

• Test every (possible) rule in the decision table.• Recommended for logically complex situations.


Decision Tables

• Represent complex conditional behavior.• Support extensive analysis

– Consistency– Completeness– Redundancy– Algebraic simplification

• Executable (and compilable)• Two forms: Limited and Extended Entry.• “Don't Care” condition entries require special

attention.• Dependencies usually yield impossible situations


Decision Table TerminologyStub Entry

Conditions

Actions

c1 c2 c3

a1 a2 a3 a4

T FTrue False

--

X X

X X

X

X X

X X

X

T F --

True

True

False

False

Rule


One Decision Table for the Triangle Problem

XXXa5: Impossible

Xa4: Equilateral

XXXa3: Isosceles

Xa2: Scalene

Xa1: Not a triangle

NYNYNYNY--c4: b = c?

NNYYNNYY--c3: a = c?

NNNNYYYY--c2: a = b?YYYYYYYYNc1: a, b, c are a triangle?

Why are rules 3, 4, and 6 impossible?


Decision Table with MutuallyExclusive Conditions

T——c3: February

—T—c2: 31-day month

——Tc1: 30-day month

Rule 3Rule 2Rule 1Conditions


conditions R1 R2 R3 c1: month in M1 T -- -- c2: month in M2 -- T -- c3: month in M3 -- -- T Rule count 4 4 4

Rule Counting to Check for Completeness

• A limited Entry decison table with n conditions has 2n rules • A Don't Care entry doubles the count of a rule • What are the possibilities when rule count is not 2n ? • More precise to use F! (must be false) than -- (don't care)


A Redundant Decision Table

•! Rule 9 is identical to Rule 4 (T, F, F)

•! Since the action entries for rules 4 and 9 are identical, ! there is no ambiguity, just redundancy.

conditions! 1-4! 5! 6! 7! 8! 9! c1: !! T! F! F! F! F! T

! c2: !! --! T! T! F! F! F

! c3: !! --! T! F! T! F! F

! a1:! ! X! X! X! --! --! X! !

! a2:! ! --! X! X! X! --! --

! a3:! ! X! --! X! X! X! X


Decision Table Exercise(revisit the false negative, false positive question)

• Suggested conditions– Expected output is correct– Observed output is correct– Expected and observed outputs agree

• Suggested actions– True pass– True fail– False pass– False fail– Impossible


An Inconsistent Decision Table

• Rule 9 is identical to Rule 4 (T, F, F) • Since the action entries for rules 4 and 9 are different there is ambiguity. • This table is inconsistent, and the inconsistency implies non-determinism.

conditions 1-4 5 6 7 8 9 c1: T F F F F T c2: -- T T F F F c3: -- T F T F F a1: X X X -- -- -- a2: -- X X X -- X a3: X -- X X X --


Nextdate Limited Entry Decision Table

Conditions c1: month in M1? c2: month in M2? c3: month in M3? c4: day in D1? c5: day in D2? c6: day in D3? c7: day in D4? c8: leap year? Actions a1: impossible a2: next date

This decision table will have 256 rules,many of which will be logically impossible.


Decision Table Based Test Cases1. Decision table testing begins with equivalence classes forconditions as in equivalence class testing.

2. The sparseness due to the assumption of independence isaddressed by careful examination of elements in the cross product.

3. For the equivalence classes defined earlier, the cross productcontains 36 elements. The corresponding decision table has 36 rules. <M1, D1, Y1>, <M1, D2, Y1>, <M1, D3, Y1>, <M1, D4, Y1>, <M2, D1, Y1>, <M2, D2, Y1>, <M2, D3, Y1>, <M2, D4, Y1>, <M3, D1, Y1>, <M3, D2, Y1>, <M3, D3, Y1>, <M3, D4, Y1>, <M1, D1, Y2>, <M1, D2, Y2>, <M1, D3, Y2>, <M1, D4, Y2>, <M2, D1, Y2>, <M2, D2, Y2>, <M2, D3, Y2>, <M2, D4, Y2>, <M3, D1, Y2>, <M3, D2, Y2>, <M3, D3, Y2>, <M3, D4, Y2>, <M1, D1, Y3>, <M1, D2, Y3>, <M1, D3, Y3>, <M1, D4, Y3>, <M2, D1, Y3>, <M2, D2, Y3>, <M2, D3, Y3>, <M2, D4, Y3>, <M3, D1, Y3>, <M3, D2, Y3>, <M3, D3, Y3>, <M3, D4, Y3>.

4. Notice that many of these are impossible, e.g., <M1, D4, * >


NextDate Extended Entry Decision TableConditions c1: month in M1? M2? M3? c2: day in D1? D2? D3? D4? c3: year in Y1? Y2? Y3? Actions a1: impossible a2: increment day a3: reset day a4: increment month a5: reset month a6: increment year

This decision table will have 36 rules, and corresponds to the cross product. Many of the rules will be logically impossible. Many rules would collapse, except for considerations for December.


Revised NextDate Domain Equivalence Classes

The corresponding decision table of these contains 40 elements.

•Month:–M1 = { month : month has 30 days}–M2 = { month : month has 31 days except December}–M3 = { month : month is December}–M4 = {month : month is February }

•Day–D1 = {day : 1 <= day <= 27}–D2 = {day : day = 28 }–D3 = {day : day = 29 }–D4 = {day : day = 30 }–D5 = {day : day = 31 }

•Year (are these disjoint?)–Y1 = {year : year is a leap year}–Y2 = {year : year is a common year}


Xa6: increment year

XXXa5: reset month

XXXXa4: incrementmonth

XXXXa3: reset day

XXXXXXXXXXXXXXXa2: increment day

Xa1: impossible

-Y2

Y1

Y2

Y1

-------------c3: year in

D5

D4

D3

D3

D2

D2

D1

D5

D4

D3

D2

D1,

D5

D4

D3

D2

D1

D5

D4

D3

D2

D1

c2: day in

M4M3M2M1c1: month in

22212019181716151413121110987654321

NextDate Extended Entry Decision Table

Notice there are 40 rules in this decision table, corresponding tothe 40 elements in the cross product of the revised equivalenceclasses.


Xa6:incrementyear

Xa5: resetmonth

XXXXa4:incrementmonth

XXXXXa3: reset day

XXXXXa2:incrementday

XXXa1:impossible

-Y2Y1Y2Y1--------c3: year in

D4, D5D3D3D2D2D1D5D1,D2,D3,D4D5D1,D2,D3,D4D5D4D1,D2,D3c2: day in

M4M3M2M1c1: month in

21,2220191817161511-14106-9541-3rules

NextDate Extended Entry Decision TableAlgebraically Condensed to 13 rules (test cases)


Procedure for Decision Table Based Testing

1. Determine conditions and actions. (Might need to iterate) 2. Develop a (the!) Decision Table, watching for • completeness • don't care entries • redundant and inconsistent rules 3. Each rule defines a test case.


Procedure for Decision Table Based Testing

• Determine conditions and actions. (Might need toiterate)

• Develop a (the!) Decision Table, watching for– Completeness– Don’t care entries– Redundant and/or inconsistent entries– Impossible rules

• Each rule defines a test case.

Software Testing: A Craftsman’s Approach, 3rd Edition Retrospective on Functional Testing

Chapter 8

Retrospective on FunctionalTesting


Retrospective on FunctionalTesting

• Test case development effort

• Test case effectiveness

• Test method selection guidelines

• Case study


Test Case Development Effort

• As with so many things in life,“You get out of it what you put into it.” --Dad

• Boundary value: almost mechanical

• Equivalence class: effort to identify classes

• Decision table: still more effort


Numbers of Test Cases

boundary value

decision table

low

high

Number of Test Cases

Sophistication

equivalence class


Test Case Development Effort

Effort to Identify Test Cases

boundary value

decision table

low

high

Sophistication

equivalence class


Test Case Effectiveness

• True trade-off between development effort andnumber of test cases.

• Vulnerabilities– Boundary value testing has gaps and redundancies, and

many test cases.– Equivalence class testing eliminates the gaps and

redundancies, but cannot deal with dependenciesamong variables.

– Decision table testing extends equivalence class testingby dealing with dependencies, and supports algebraicreduction of test cases.


Appropriate Choices of Test Methods

xxa9. decision table

xxa8. strong robust equiv. class

xxa7. strong normal equiv. class

xxa6. weak robust equiv. class

xxa5. weak normal equiv. class

a4. robust worst case BVA

a3. worst case BVA

xa2. robustness BVA

xa1. boundary value analysis (BVA)

–NYNY–NYNYc4. exception handling?

–NNYY–NNYYc3. single fault assumption?

NYYYYNYYYYc2. independent variables?

LLLLLPPPPPc1. variables (P, physical, L, logical)


Case StudyA hypothetical Insurance Premium Program computes the semi-annual car insurance premium based on two parameters: the policyholder's age and driving record:

Premium = BaseRate*ageMultiplier – safeDrivingReduction

The ageMultiplier is a function of the policy holder's age, and thesafe driving reduction is given when the current points (assigned bytraffic courts for moving violations) on the policy holder's driver'slicense are below an age-related cutoff. Policies are written fordrivers in the age range of 16 to 100. Once a policy holder has 12points, his/her driver's license is suspended (hence there is no needfor insurance). The BaseRate changes from time to time; for thisexample, it is $500 for a semi-annual premium.


Insurance Premium Program Data

20051.560<= age <= 100

15070.845<= age < 60

10051.035<= age < 45

5031.825<= age < 35

5012.816<= age < 25

Safe DrivingReduction

PointsCutoff

AgeMultiplier

Age Range


Insurance Premium ProgramCalculations, Test Method Selection

• Premium = BaseRate*ageMultiplier – safeDrivingReduction• ageMultiplier = F1(age) [from table]• safeDrivingReduction = F2(age, points) [from table]• age and safeDrivingReduction are physical variables, with

a dependency in F2.• Boundary values for age: 16, 17, 54, 99, 100• Boundary values for safeDrivingReduction: 0, 1, 6, 11, 12• Robust values for age and safeDrivingReduction are not

allowed by business rules.• Worst case BVA yields 25 test cases, and many gaps,

some redundancy. Need something better.


Graph of Boundary Value Test Cases

20 100806040age

15

10

5

0

points

Severe gaps!


Graph of Boundary Value Test Cases(refined boundaries)

20 100806040age

15

10

5

0

points

Severe redundancy!


Insurance Premium Program TestMethod Selection

• age has ranges that receive similar treatment.equivalence class testing is indicated.– Age ranges per the data table

• The points cutoff is also a range, furtherindication for equivalence class testing.– Points {0, 1}– Points {2, 3}– Points {4, 5}– Points {6, 7}– Points {8, 9, 10, 11, 12}


Insurance Premium Program StrongNormal Equivalence Class Test Cases

20 100806040age

15

10

5

0

points

Still a lot of redundancy, try decision tables.


Insurance Premium Program DecisionTable Test Cases

--200--150--100--50--50a2. safeDriving

1.51.50.80.81.81.81.81.82.82.8a1. agemultiplier

5-120-47-120-65-120-43-130-21-120c2. points60-10045-6035-4525-3516-25c1. age is


Insurance Premium Program DecisionTable Test Cases

20 100806040age

15

10

5

0

points

What about age range endpoints?


Insurance Premium Program Test Cases(Decision table with boundary values hybrid)

20 100806040age

15

10

5

0

points

Ahhhh, at last!


Wrap Up

• The inherent nature of the program beingtested should dictate the test method.– The decision table “expert system” (slide 7)

recommendation is just a start.– Applications are seldom “chemically pure”.

• Hybrid combinations of test methods can bevery useful.

• Good judgment, based on insight, is a signof a craftsperson.

Software Testing: A Craftsman’s Approach, 3rd Edition Path Testing II

Chapter 9

Path Testing–Part 2


(McCabe) Basis Path Testing

• in math, a basis "spans" an entire space, such that everything inthe space can be derived from the basis elements.

• the cyclomatic number of a strongly connected directed graph isthe number of linearly independent cycles.

• given a program graph, we can always add an edge from the sinknode to the source node to create a strongly connected graph.(assuming single entry, single exit)

• computing V(G) = e - n + p from the modified program graphyields the number of independent paths that must be tested.

• since all other program execution paths are linear combinationsof the basis path, it is necessary to test the basis paths. (Somesay this is sufficient; but that is problematic.)

• the next few slides follow McCabe's original example.


McCabe's ExampleDerived, StronglyConnected Graph

A

B

C

D

E

F

G

1 2

3 45 6

78

9 10

A

B

C

D

E

F

G

V(G) = 10 - 7 + 2(1) = 5

V(G) = 11 - 7 + 1 = 5

McCabe's OriginalGraph


McCabe's Baseline Method

To determine a set of basis paths, 1.

Pick a "baseline" path that corresponds to normal execution. (The baseline should have as many decisions as possible.)

2.

To get succeeding basis paths, retrace the baseline until you reach a decision node. "Flip" the decision (take another alternative) and continue as much of the baseline as possible.

3.

Repeat this until all decisions have been flipped. When you reach V(G) basis paths, you're done.

4.

If there aren't enough decisions in the first baseline path, find a second baseline and repeat steps 2 and 3.

Following this algorithm, we get basis paths for McCabe's example.


path \ edges traversed 1 2 3 4 5 6 7 8 9 10 p1: A, B, C, G 1 0 0 1 0 0 0 0 1 0 p2: A, B, C, B, C, G 1 0 1 2 0 0 0 0 1 0 p3: A, B, E, F, G 1 0 0 0 1 0 0 1 0 1 p4: A, D, E, F, G 0 1 0 0 0 1 0 1 0 1 p5: A, D, F, G 0 1 0 0 0 0 1 0 0 1 ex1: A, B, C, B, E, F, G 1 0 1 1 1 0 0 1 0 1 ex2: A, B, C, B, C, B, C, G 1 0 2 3 0 0 0 0 1 0 ex1 = p2 + p3 - p1 ex2 = 2p2 - p1

Basis Paths

A

B

C

D

E

F

G

1 2

3 45 6

78

9 10


McCabe Basis Paths in the Triangle Program

V(G) = 23 - 20 + 2(1) = 5

Basis Path Set B1p1: 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 18, 19, 20, 22, 23 (mainline)p2: 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 16, 18, 19, 20, 22, 23 (flipped at 9)p3: 4, 5, 6, 7, 8, 9, 11, 12, 13, 21, 22, 23 (flipped at 13)p4: 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 20, 22, 23 (flipped at 14)p5: 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 16, 17, 19, 20, 22, 23 (flipped at 16)

There are 8 topologically possible paths.4 are feasible, and 4 are infeasible.

Exercise: Is every basis path feasible?

13

14

15 16

17 18

19

20

21

22

4 5 6 7 8

10

9

11

12

23


Essential Complexity• McCabe’s notion of Essential Complexity deals with the

extent to which a program violates the precepts ofStructured Programming.

• To find Essential Complexity of a program graph,– Identify a group of source statements that corresponds to one of the

basic Structured Programming constructs.– Condense that group of statements into a separate node (with a new

name)– Continue until no more Structured Programming constructs can be

found.– The Essential Complexityof the original program is the cyclomatic

complexity of the resulting program graph.• The essential complexity of a Structured Program is 1.• Violations of the precepts of Structured Programming

increase the essential complexity.


Essential Complexity ofSchach’s Program Graph

B

DC E

4

first

A

B

DC E

F

G

last

Thisoriginal

Reducesto this

V(G) = 8 – 5 + 2(1) = 5Essential complexity is 5


A

B C

D

E

F G

H I

J K

L

first

last

aa

E

F G

H I

J K

L

first

last

b

a

H I

J KL

first

last

b

c

a

HL

first

last

b

c

da

L

first

last

d

e

first

e

Condensation with Structured Programming Constructs


Branching into a loop

Branching out of a loop

Branching into a decision

Branching out of a decision

Violations of Structured Programming Precepts


Cons and Pros

• Issues– Linear combinations of execution paths are counter-

intuitive. What does 2p2 – p1 really mean?– How does the baseline method guarantee feasible basis

paths?– Given a set of feasible basis paths, is this a sufficient test?

• Advantages– McCabe's approach does address both gaps and

redundancies.– Essential complexity leads to better programming

practices.– McCabe proved that violations of the structured

programming constructs increase cyclomatic complexity,and violations cannot occur singly.


Program NextDate Dim tomorrowDay,tomorrowMonth,tomorrowYear As IntegerDim day,month,year As Integer1. Output ("Enter today's date in the form MM DD YYYY")2. Input (month,day,year)3. Case month Of4. Case 1: month Is 1,3,5,7,8,Or 10: '31 day months (except Dec.)5. ! If day < 316. ! ! Then tomorrowDay = day + 17. ! ! Else8. ! ! ! tomorrowDay = 19. ! ! ! tomorrowMonth = month + 110. ! EndIf11. Case 2: month Is 4,6,9,Or 11 '30 day months12. ! If day < 3013. ! ! Then tomorrowDay = day + 114. ! ! Else15. ! ! ! tomorrowDay = 116. ! ! ! tomorrowMonth = month + 117. ! EndIf18. Case 3: month Is 12: 'December19. ! If day < 31 20. ! ! Then tomorrowDay = day + 121. ! ! Else22. ! ! ! tomorrowDay = 123. ! ! ! tomorrowMonth = 124. ! ! ! If year = 201225. ! ! ! ! Then Output ("2012 is over")26. ! ! ! ! Else tomorrow.year = year + 127. ! ! ! EndIf28. ! EndIf


29. Case 4: month is 2: 'February30. ! If day < 2831. ! ! Then tomorrowDay = day + 132. ! ! Else!33. ! ! ! If day = 2834. ! ! ! ! Then!35. ! ! ! ! ! If ((year MOD 4)=0)AND((year MOD 400)"0)36. ! ! ! ! ! ! Then tomorrowDay = 29 'leap year37. ! ! ! ! ! ! Else! ! 'not a leap year38. ! ! ! ! ! ! ! tomorrowDay = 139. ! ! ! ! ! ! ! tomorrowMonth = 340. ! ! ! ! ! EndIf41. ! ! ! ! Else! If day = 2942. ! ! ! ! ! ! ! Then tomorrowDay = 143. ! ! ! ! ! ! ! ! tomorrowMonth = 344. ! ! ! ! ! ! ! Else! Output("Cannot have Feb.", day)45. ! ! ! ! ! ! EndIf46. ! ! ! EndIf47. ! EndIf48. EndCase49. Output ("Tomorrow's date is", tomorrowMonth,! ! ! ! tomorrowDay, tomorrowYear)50. End NextDate


11

1

2

3

4

5

6

7

8

9

10

12

13

14

15

16

17

18

19

20

21

22

23

24

25 26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

50

47

48

49

44

45

46


15 Output("Locks sold: ", totalLocks)16 Output("Stocks sold: ", totalStocks)17 Output("Barrels sold: ", totalBarrels)18 sales = lockPrice*totalLocks + stockPrice*totalStocks + barrelPrice * totalBarrels19 Output("Total sales: ", sales)20 If (sales > 1800.0) 21 Then22 commission = 0.10 * 1000.023 commission = commission + 0.15 * 800.024 commission = commission + 0.20*(sales-1800.0)25 Else If (sales > 1000.0)26 Then27 commission = 0.10 * 1000.028 commission = commission + 0.15*(sales-1000.0)29 Else commission = 0.10 * sales30 EndIf31 EndIf32 Output("Commission is $", commission)33 End Commission

Program Commission Dim lockPrice, stockPrice, barrelPrice As Real Dim locks, stocks, barrels As Integer Dim totalLocks, totalStocks As Integer Dim totalBarrels As Integer Dim lockSales, stockSales As Real Dim barrelSales As Real Dim sales, commission As Real1 lockPrice = 45.02 stockPrice = 30.03 barrelPrice = 25.04 totalLocks = 05 totalStocks = 06 totalBarrels = 07 Input(locks)8 While NOT(locks = -1)9 Input(stocks, barrels)10 totalLocks = totalLocks + locks11 totalStocks = totalStocks + stocks12 totalBarrels = totalBarrels + barrels13 Input(locks)14 EndWhile

Commission Program Pseudo-Code


11

1 2 3 4 5 6 7

8 9 10 12 13

14 15 16 17 18 19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

Software Testing: A Craftsman’s Approach, 3rd Edition Data Flow Testing

Chapter 10

Data Flow TestingSlice Testing


Data Flow Testing

• Often confused with "dataflow diagrams“.• Main concern: places in a program where data

values are defined and used.• Static (compile time) and dynamic (execution

time) versions.• Static: Define/Reference Anomalies on a

variable that– is defined but never used (referenced)– is used but never defined– is defined more than once

• Starting point is a program, P, with program graphG(P), and the set V of variables in program P.

• "Interesting" data flows are then tested asmini-functions.


Definitions• Node n ∈ G(P) is a defining node of the

variable v ∈ V, written as DEF(v, n), iff thevalue of the variable v is defined at thestatement fragment corresponding to node n.

• Node n ∈ G(P) is a usage node of the variablev ∈ V, written as USE(v, n), iff the value of thevariable v is used at the statement fragmentcorresponding to node n.

• A usage node USE(v, n) is a predicate use(denoted as P-use) iff the statement n is apredicate statement; otherwise, USE(v, n) is acomputation use (denoted C-use).


More Definitions

• A definition-use path with respect to a variable v(denoted du-path) is a path in PATHS(P) suchthat for some v ∈ V, there are define and usagenodes DEF(v, m) and USE(v, n) such that m and nare the initial and final nodes of the path.

• A definition-clear path with respect to a variablev (denoted dc-path) is a definition-use path inPATHS(P) with initial and final nodes DEF (v, m)and USE (v, n) such that no other node in thepath is a defining node of v.


Example: first part of the Commission Program1. Program Commission (INPUT,OUTPUT)2. Dim locks, stocks, barrels As Integer3. Dim lockPrice, stockPrice, barrelPrice As Real4. Dim totalLocks, totalStocks, totalBarrels As Integer5. Dim lockSales, stockSales, barrelSales As Real6. Dim sales, commission As Real7. lockPrice = 45.08. stockPrice = 30.09. barrelPrice = 25.010. totalLocks = 011. totalStocks = 012. totalBarrels = 013. Input(locks)14. While NOT(locks = -1)15. Input(stocks, barrels)16. totalLocks = totalLocks + locks17. totalStocks = totalStocks + stocks18. totalBarrels = totalBarrels + barrels19. Input(locks)20. EndWhile21. Output(“Locks sold: “, totalLocks)22. Output(“Stocks sold: “, totalStocks)23. Output(“Barrels sold: “, totalBarrels)


Rest of Commission Problem23. Output(“Barrels sold: “, totalBarrels)24. lockSales = lockPrice * totalLocks25. stockSales = stockPrice * totalStocks26. barrelSales = barrelPrice * totalBarrels27. sales = lockSales + stockSales + barrelSales28. Output(“Total sales: “, sales)29. If (sales > 1800.0)30. Then31. commission = 0.10 * 1000.032. commission = commission + 0.15 * 800.033. commission = commission + 0.20 *(sales-1800.0)34. Else If (sales > 1000.0)35. Then36. commission = 0.10 * 1000.037. commission = commission + 0.15 *(sales-1000.0)38. Else39. commission = 0.10 * sales40. EndIf41. EndIf42. Output(“Commission is $”, commission)43. End Commission


Program Graph of Commission Problem107 8 9 11 12 13

14

15 16 2017 18 19

21 22 23 24 25 26 27 28

29

31

32

33

36

41 42

30 34

38

35

37

39

40 43


Define/Use Test Cases

• Technique: for a particular variable,– find all its definition and usage nodes, then– find the du-paths and dc-paths among these.– for each path, devise a "suitable" (functional?) set of test cases.

• Note: du-paths and dc-paths have both static and dynamicinterpretations– Static: just as seen in the source code– Dynamic: must consider execution-time flow (particularly

for loops)• Definition clear paths are easier to test

– No need to check each definition node, as is necessary fordu-paths


Define and Use Nodes

18, 23, 26 12, 18totalBarrels

17, 22, 25 11, 17totalStocks

16, 21, 24 10, 16totalLocks

1815barrels

1715stocks

14, 16 13, 19locks

Used at Node Defined at NodeVariable


Example (continued)13. Input(locks)14. While NOT(locks = -1)15. Input(stocks, barrels)16. totalLocks = totalLocks + locks17. totalStocks = totalStocks + stocks18. totalBarrels = totalBarrels + barrels19. Input(locks)20. EndWhileWe have• DEF (locks, 13), DEF (locks, 19)• USE (locks, 14), a predicate use• USE (locks, 16). A computation use• du-paths for locks are the node sequences <13, 14> (a dc-path),

<13, 14, 15, 16>, <19, 20, 14 >, < 19, 20, 14 , 15, 16>• Is <13, 14, 15, 16> definition clear?• Is < 19, 20, 14, 15, 16> definition clear? What about repetitions?


Coverage Metrics Based on du-paths• In the following definitions, T is a set of paths in

the program graph G(P) of a program P, withthe set V of variables.

• The set T satisfies the All-Defs criterion for theprogram P iff for every variable v ∈ V, Tcontains definition-clear paths from everydefining node of v to a use of v.

• The set T satisfies the All-Uses criterion for theprogram P iff for every variable v ∈ V, Tcontains definition-clear paths from everydefining node of v to every use of v, and to thesuccessor node of each USE(v, n).


Coverage Metrics Based on du-paths(continued)

• The set T satisfies the All-P-Uses/Some C-Usescriterion for the program P iff for every variablev ∈ V, T contains definition-clear paths fromevery defining node of v to every predicate useof v; if a definition of v has no P-uses, adefinition-clear path leads to at least onecomputation use.


Coverage Metrics Based on du-paths(continued)

• The set T satisfies the All-C-Uses/Some P-Usescriterion for the program P iff for every variablev ∈ V, T contains definition-clear paths fromevery defining node of v to every computationuse of v; if a definition of v has no C-uses, adefinition-clear path leads to at least onepredicate use.


Coverage Metrics Based on du-paths(concluded)

• The set T satisfies the All-du-paths criterion forthe program P iff for every variable v ∈ V, Tcontains definition-clear paths from everydefining node of v to every use of v and to thesuccessor node of each USE(v, n), and thatthese paths are either single-loop traversals orcycle-free.


Rapps-Weyuker Coverage SubsumptionAll Paths

All DU Paths

All UsesAll C Uses Some P Uses

All P Uses Some C Uses

All Defs All P Uses

All Edges

All Nodes

S. Rapps and E. J. Weyuker "Selecting Software Test Data Using Data Flow Information" IEEE Transactions of Software Engineering , vol 11 no 4 IEEE Computer Society Press, Washington, D. C. , April 1985, pp 367 - 375.


Exercise: Where does the “Alldefinition-clear paths” coverage metricfit in the Rapps-Weyuker lattice?


Data Flow Testing Strategies

• Data flow testing is indicated in– Computation-intensive applications– “long” programs– Programs with many variables

• A definition-clear du-path represents a smallfunction that can be tested by itself.

• If a du-path is not definition-clear, it shouldbe tested for each defining node.


Slice Testing• Often confused with "module execution paths"• Main concern: portions of a program that

"contribute" to the value of a variable at somepoint in the program.

• Nice analogy with history -- a way to separate acomplex system into "disjoint" components thatinteract:– European history– North American history– Orient history

• A dynamic construct.


Slice Testing Definitions

Starting point is a program, P, with program graphG(P), and the set V of variables in program P.Nodes in the program graph are numbered andcorrespond to statement fragments.

• Definition: The slice on the variable set V atstatement fragment n, written S(V, n), is the setof node numbers of all statement fragments inP prior to n that contribute to the values ofvariables in V at statement fragment n.

• This is actually a “backward slice”.• Exercise: define a “forward slice”.


Fine Points• "prior to" is the dynamic part of the definition.• "contribute" is best understood by extending the

Define and Use concepts:– P-use: used in a predicate (decision)– C-use: used in computation– O-use: used for output– L-use: used for location (pointers, subscripts)– I-use: iteration (internal counters, loop indices)– I-def: defined by input– A-def: defined by assignment

• usually, the set V of variables consists of just oneelement.

• can choose to define a slice as a compilable set ofstatement fragments -- this extends the meaning of"contribute"

• because slices are sets, we can develop a latticebased on the subset relationship.


In the program fragment13. Input(locks)14. While NOT(locks = -1)15. Input(stocks, barrels)16. totalLocks = totalLocks + locks17. totalStocks = totalStocks + stocks18. totalBarrels = totalBarrels + barrels19. Input(locks)20.EndWhile

There are these slices on locks (notice thatstatements 15, 17, and 18 do not appear):

S1: S(locks, 13) = {13}S2: S(locks, 14) = {13, 14, 19, 20}S3: S(locks, 16) = {13, 14, 19, 20}S4: S(locks, 19) = {19}


Lattice of Slices

• Because a slice is a set of statement fragmentnumbers, we can find slices that are subsets ofother slices.

• This allows us to “work backwards” frompoints in a program, presumably where a faultis suspected.

• The statements leading to the value ofcommission when it is output are an excellentexample of this pattern.

• Some researchers propose that this is the waygood programmers think when they debugcode.


Example Lattice of Slices

S39

S34

S35

S36

S37

S40

S38

S34: S(commission, 41) = {41}

S35: S(commission, 42) = {41, 42}

S36: S(commission, 43) = {3, 4, 5, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 30, 36, 41, 42, 43}

S37: S(commission, 47) = {47}

S38: S(commission, 48) = {3, 4, 5, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 36, 47, 48}

S39: S(commission, 50) = {3, 4, 5, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 36, 50}

S40: S(commission, 51) = {3, 4, 5, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 36, 41, 42, 43, 47, 48, 50}


Diagnostic Testing with Slices

• Relative complements of slices yield a "diagnostic"capability. The relative complement of a set B with respectto another set A is the set of all elements of A that are notelements of B. It is denoted as A - B.

• Consider the relative complement set S(commission, 48) -S(sales, 35):– S(commission, 48) = {3, 4, 5,36,18,19, 20, 23, 24, 25, 26, 27, 34, 38,

39, 40, 44,45,47}– S(sales, 35) = {3, 4, 5, 36, 18, 19, 20, 23, 24, 25, 26, 27}– S(commission, 48) - S(sales, 35) = {34, 38, 39, 40, 44,45,47}

• If there is a problem with commission at line 48, we candivide the program into two parts, the computation ofsales at line 34, and the computation of commissionbetween lines 35 and 48. If sales is OK at line 34, theproblem must lie in the relative complement; if not, theproblem may be in either portion.


Programming with Slices

• One researcher suggests the possibility of“slice splicing”:– Code a slice, compile and test it.– Code another slide, compile and test it, then splice

the two slices.– Continue until the whole program is complete.

• Exercise: in what ways is slice splicingdistinct from agile (bottom up) programming?


Exercise/Discussion:When should testing stop?

• when you run out of time?• when continued testing causes no new

failures?• when continued testing reveals no new

faults?• when you can't think of any new test cases?• when you reach a point of diminishing

returns?• when mandated coverage has been attained?• when all faults have been removed?

Software Testing: A Craftsman’s Approach, 3rd Edition Retrospective on Structural Testing

Chapter 11

Retrospective on Structural Testing


Structural Testing Comparison

• How much testing is enough?

• Effort and size trendlines

• Metrics for test method comparison


Exercise/Discussion:When should testing stop?

• when you run out of time?• when continued testing causes no new

failures?• when continued testing reveals no new

faults?• when you can't think of any new test cases?• when you reach a point of diminishing

returns?• when mandated coverage has been attained?• when all faults have been removed?


Number of Test Coverage Items

low

high

SophisticationDU-Path

DD-Path

Basis Path

Slice


Effort to Identify Test Coverage Items

low

high

Sophistication

DU-PathDD-Path

Basis Path

Slice


Number of Coverage Items in theCommission Problem

BasisDD-Path DU-Path Slices048

1216202428323640


Metrics for Test Method Comparison• Assume that a functional testing technique M

generates m test cases, and that these testcases are tracked with respect to a structuralmetric S that identifies s elements in the unitunder test. When the m test cases are executed,they traverse n of the s structural elements.

• This framework supports the definition ofmetrics for testing effectiveness.


Metrics for Testing Effectiveness

• The coverage of a methodology M withrespect to a metric S is ratio of n to s. Wedenote it as C(M,S).

• The redundancy of a methodology M withrespect to a metric S is ratio of m to s. Wedenote it as R(M,S).

• The net redundancy of a methodology M withrespect to a metric S is ratio of m to n. Wedenote it as NR(M,S).


Sample Comparisons

0.630.631.00404025Slices0.760.761.00333325DU-Path2.272.271.00111125DD-Path0.270.271.0011113Decision Table2.272.271.00111125Output BVA

CommissionProgram

11.3611.361.001111125Worst Case BVA2.141.360.6411715BVA

Triangle ProgramNR(M,S)R(M,S)C(M,S)snmMethod

Software Testing: A Craftsman’s Approach, 3rd Edition Levels of Testing

Chapter 12

Levels of Testing


Levels and Life Cycle Models

• Levels of testing depend primarily on thesoftware life cycle used.

• BUT, most forms of testing levels are derivedfrom the V-Model version of the good, oldWaterfall Model.

• Iterative models introduce the need forregression testing.

• System testing is greatly enhanced when anexecutable specification is used.


Software Development Life Cycle Models

• Waterfall (presumes perfect foresight)• Incremental (delayed prototypes)• Rapid Prototyping (elicit user feedback)• Operational Specification (executable)• Transformational Implementation (lab only)• Agile Methods

Increasingly Operationalviews

Ref. Agresti, New Paradigms of Software DevelopmentIEEE Tutorial


The Waterfall ModelRequirements Specification

Preliminary Design

Detailed Design

Coding

Unit Testing

Integration Testing

System Testing

Output of a phase is the inputto the next phase


Requirements Specification

Preliminary Design

Detailed Design

Coding

Unit Testing

Integration Testing

System Testing

The Waterfall Model (continued)

Feedback Cycles

what

what

what

how

how

how


The Waterfall Model (aka the V-Model)


Preliminary Design

Detailed Design

Coding

Unit Testing

Integration Testing

System Testing

Levels of Abstraction and Testing


The Waterfall Model: Advantages

• Framework fits well with– levels of abstraction– levels of management– programming language packaging

• Phases have clearly defined end products• Convenient for project management• Works well with Functional Decomposition• Massive parallel development


The Waterfall Model: Disadvantages

• Long customer feedback cycle resolving faults found duringsystem testing is extremely expensive

• "Exists for the convenience of management"(-- M. Jackson) stifles creativity and unnecessarily constraintsdesigner's thought processes

• Stresses analysis to the exclusion of synthesis• High peak in manpower loading profile• "Requires perfect foresight" -- William Agresti

any errors, omissions in early phases will propagate



Preliminary Design

Series of Builds

Detailed Design

Coding

Unit Testing

Integration Testing

Regression Testing

Progression Testing

Incremental Software Development


Series of Prototypes

Preliminary Design

Detailed Design

Coding

Integration Testing

System Testing

Build Prototype

Define Prototype Objectives

Customer Feedback

Unit Testing

Rapid Prototyping Software Development


Rapid Prototyping

Why prototype? 1. To determine feasibility 2. To obtain early customer feedback Keep or dispose? To be rapid, many compromises are made. If a prototype is kept, it will be extremely difficult to modify and maintain. Best practice: dispose once purpose has been served.


execute spec

Develop executable

specification

Customer Feedback


Preliminary Design

Detailed Design

Coding

Integration Testing

System Testing

Unit Testing

Software Development with an Executable Specification


Executable Specifications

Why use an executable specification? 1. To determine behavior 2. To obtain early customer feedback Other uses? 1. Automatic generation of system test cases. 2. Develop order of test case execution 3. Training 4. Early analysis


Transformational Implementation


System Testing

Formal Requirements Specification

Customer Testing

Working System

Series of Transformations


Transformational Specification Pros and Cons

• Advantages:– Intermediate Waterfall phases (design, code, test)

are eliminated.– Customer tests delivered system– All maintenance is done on specification

• Disadvantages:– Specification must be very formal (predicate

calculus)– Series of transformations is not well understood;

tends to be specific to application domains– Very limited success so far.– Doesn't scale up well.– Knowing what to change in the formal specification

is difficult


Agile Methods

• Many flavors– eXtreme Programming (XP)– SCRUM– Test-Driven Development

• Customer-Driven• Goals

– Respond to customer– Reduce unnecessary effort– Always have something that works


Length of Feedback Cycle(Customer/Developer)

Waterfall

Agile

Where would you putthe other life cyclemodels?


Hybrid Models

• Because they replace the Requirementsspecification phase, rapid prototyping andexecutable specifications can be merged withthe iterative models.

• The Agile Models are highly iterative, but theytypically are not used in combination with rapidprototyping or executable specifications.

Integra(on tes(ng

by Kris(an Sandahl IDA

TDDD04 spring 2012

2

Levels of tes(ng

Requirements

System Design (Architecture,

High-level Design)

Module Design (Program Design, Detailed Design)

Implementation of Units (classes, procedures,

functions)

Unit testing

Module Testing (Integration testing of units)

System Testing (Integration testing of modules)

Acceptance Test

(Release testing)

Validate Requirements, Verify Specification

Verify System Design

Verify Module Design

Verify Implementation

Project Management, Software Quality Assurance (SQA), Supporting Tools, Education

Maintenance

Outline

•  Informal example •  Theory:

– Decomposi(on-‐based integra(on – Call-‐graph based integra(on – Path-‐based integra(on

Example: Local bus card reader Sell (ckets Registrer travel

Chose (cket

Display

Read RFID

Check balance Check validity

Communicate with server

Deduct money

Power supply

Show balance

User buNons Func(on group, aka Capability, aka Anatom

and

Dependency

Example: Organic integra(on plan Sell (ckets Registrer travel

Chose (cket

Display

Read RFID

Check balance Check validity


Deduct money

Power supply

Show balance

User buNons

Services

User interface

Server func(ons

Communica(on

Hardware and supply

Big Bang integra(on tes(ng Sell (ckets

Registrer travel

Chose (cket

Display

Read RFID Check balance Check validity

Communicate with server Deduct money

Power supply

Show balance

User buNons Bring everything

together Switch on the current Try to buy a (cket

!

Is Big Bang smart? •  Perhaps for small systems •  Well-‐defined components and interfaces •  No extra soWware needed Problems: •  It is very difficult to isolate the defects found, as it is very difficult

to tell whether defect is in component or interface. •  Defects present at the interfaces of components are iden(fied at

very late stage. •  There is high probability of missing some cri(cal defects which

might surfaced in produc(on. •  It is very difficult to make sure that all the cases for integra(on

tes(ng are covered. (www.Tes(ngGeek.com)

BoNom-‐up integra(on 1

Power supply Hardware and supply

Environment crea(on Install components Draw cables Measure – compare with calcula(on

BoNom-‐up integra(on 2


Power supply

Communica(on

Hardware and supply

Rudimental client Rudimental server

Communica(on possible?

Environment

Drivers

•  A pretend module that requires a sub-‐system and passes a test case to it

black-‐box view

Driver

setup driver SUT(x) verifica(on

SUT

driver

SUT

SUT – System Under Test

Is boNom-‐up smart? •  If the basic func(ons are complicated, error-‐prone or has development risks

•  If boNom-‐up development strategy is used •  If there are strict performance or real-‐(me requirements

Problems: •  Lower level func(ons are oWen off-‐the shelf or trivial •  Complicated User Interface tes(ng is postponed •  End-‐user feed-‐back postponed •  Effort to write drivers.

Top-‐down integra(on 1 Sell (ckets Registrer travel

Chose (cket

Show balance

Services

Test end-‐users Envrionment: Modules for the services Driving a prototype interface

Is it possible to perform services in a good way?

Top-‐down integra(on 2 Sell (ckets Registrer travel

Chose (cket

Display

Read RFID

Func(on 1 Func(on 2 Func(on 2

Show balance

User buNons

Services

User interface

Rudimentary Server func(ons

Stubs Are all our test scenarios fulfilled?

Stub

•  A program or a method that simulates the input-‐output func6onality of a missing sub-‐system by answering to the decomposi(on sequence of the calling sub-‐system and returning back simulated or ”canned” data.

SUT Service(x)

Check x Stub Return y; end

SUT

Stub

Is top-‐down smart? •  Test cases are defined for func(onal requirements of the system

•  Defects in general design can be found early •  Works well with many incremental development methods

•  No need for drivers Problems: •  Technical details postponed, poten(al show-‐stoppers •  Many stubs are required •  Stubs with many condi(ons are hard to write

Decomposi(on-‐based integra(on

The func(onal decomposi(on tree: •  hierarchical order of processes •  return edges excluded •  reflects the lexical inclusion of units, in terms of the order in which they need to be compiled

Jorgensen, Paul C "Chapter 13 -‐ Integra(on Tes(ng". SoWware Tes(ng: A CraWsman's Approach, Third Edi(on. Auerbach Publica(ons. © 2008. Books24x7. Go to: hNp://guide.bibl.liu.se/datavetenskap Click: Books24x7, Login Search: ISBN:9780849374753

Example: SATM from Jorgensen

17

Func6onal Decomposi6on of the SATM System

1

A 10 B

D E 11 12 13 14 15

2 3 4 5 6 7 8 9

C

16 17 F 22

18 19 20 21 23 24 25 26 27

Table 1: SATM Units and Abbreviated Names

Unit Level Unit Name 1 1 SATM System A 1.1 Device Sense & Control D 1.1.1 Door Sense & Control 2 1.1.1.1 Get Door Status 3 1.1.1.2 Control Door 4 1.1.1.3 Dispense Cash E 1.1.2 Slot Sense & Control 5 1.1.2.1 WatchCardSlot 6 1.1.2.2 Get Deposit Slot Status 7 1.1.2.3 Control Card Roller 8 1.1.2.3 Control Envelope Roller 9 1.1.2.5 Read Card Strip 10 1.2 Central Bank Comm. 11 1.2.1 Get PIN for PAN 12 1.2.2 Get Account Status 13 1.2.3 Post Daily Transac6ons

Unit Level Unit Nam B 1.3 Terminal Sense & Control 14 1.3.1 Screen Driver 15 1.3.2 Key Sensor C 1.4 Manage Session 16 1.4.1 Validate Card 17 1.4.2 Validate PIN 18 1.4.2.1 GetPIN F 1.4.3 Close Session 19 1.4.3.1 New Transac6on Request 20 1.4.3.2 Print Receipt 21 1.4.3.3 Post Transac6on Local 22 1.4.4 Manage Transac6on 23 1.4.4.1 Get Transac6on Type 24 1.4.4.2 Get Account Type 25 1.4.4.3 Report Balance 26 1.4.4.4 Process Deposit 27 1.4.4.5 Process Withdrawal

Three level func(onal decomposi(on tree

A

C B D

E F H G

Level 1

Level 2

Level 3

Big-‐Bang tes(ng

A

C B D

E F H G

Level 1

Level 2

Level 3

Unit test A

Unit test B

Unit test H

…

System-‐wide test

Environment: A, B, C, D, E, F, G, H

BoNom-‐up tes(ng

A

C B D

E F H G

Level 1

Level 2

Level 3

Environments: Session 1: E, driver(B) S2: F, driver(B) S3: E, F, driver(B) S4: G, driver(D) S5: H, driver(D) S6: G, H, driver(D) S7: E, F, B, driver(A) S8: C, driver(A) S9: G, H, D, driver(A) S10: E, F, B, C, G, H, D, A

Number of drivers: 3 Number of sessions: 10

General formula: Number of drivers: (nodes-‐leaves) Number of sessions: (nodes-‐leaves)+edges

SATM: 10 drivers, 42 sessions

Top-‐down tes(ng

A

C B D

E F H G

Level 1

Level 2

Level 3

Environments: Session 1: A, stub(B), stub(C), stub(D) S2: A, B, stub(C), stub(D) S3: A, stub(B), C, stub(D) S4: A, stub(B), stub(C), D S5: A, B, stub(E), stub(F), C, D, stub(G), stub(H) S6: A, B, E, stub(F), C, D, stub(G), stub(H) S7: A, B, stub(E), F, C, D, stub(G), stub(H) S8: A, B, stub(E), stub(F), C, D, G, stub(H) S9: A, B, stub(E), stub(F), C, D, stub(G), H S10: A, B, E, F, C, D, G, H

Number of stubs: 7 Number of sessions: 10

General formula: Number of stubs: (nodes – 1) Number of sessions: (nodes-‐leaves)+edges

SATM: 32 stubs, 42 sessions

Sandwich tes(ng

A

C B D

E F H G

Level 1

Level 2

Level 3

Environments: Session 1: A, stub(B), stub(C), stub(D) S2: A, B, stub(C), stub(D) S3: A, stub(B), C, stub(D) S4: A, stub(B), stub(C), D S5: E, driver(B) S6: F, driver(B) S7: E, F, driver(B) S8: G, driver(D) S9: H, driver(D) S10: G, H, driver(D) S11: A, B, E, F, C, D, G, H

Number of stubs: 3 Number of drivers: 2 Number of sessions: 11

Fewer stubs and drivers Risk driven Small-‐bang at target level More complicated

Taget level

Poten(al problems

•  Ar(ficial: Assumes correct units and interfaces •  Test correct structure only •  Investment in stubs and drivers •  Retes(ng

Call-‐graph integra(on tes(ng

•  Use the call-‐graph instead of the decomposi(on tree

•  The call graph is directed •  Two types of tests:

– Pair-‐wise integra(on tes(ng – Neighborhood integra(on tes(ng

•  Matches well with development and builds •  Tests behaviour

1

5

7

20

21

9

10

12

11

16

17 18 19

22

23 24

25 26

14 2 3 6 8

4

13

15

27

Pairwise integra(on of SATM One session per edge Real code 40 sessions, but no extra code

1

5

7

20

21

9

10

12

11

16

17 18 19

22

23 24

25 26

14 2 3 6 8

4

13

15

27

Neighborhood integra(on of SATM Integra(ng direct neighbors of nodes Number of sessions: nodes – sinknodes (a sink node has no outgoing calls) SATM: 11 sessions

Poten(al problems

•  ”Small-‐bang” problems •  Especially fault isola(on in large neighborhoods

•  Restes(ng needed if a node is changed •  Assumes correct units

Path-‐based integra(on

•  Base tes(ng on system level threads •  Mo(vated by overall system behaviour, not the structure

•  Smooth prepara(on for System level tes(ng

Example: An MM-‐Path module A calls module B, which in turn calls module C

1

2

3 4

5

6

1

2

3

4

1

2 3

4

5

A B C

Defini(ons •  Defini6on: A source node in a program is a statement fragment at

which program execu(on begins or resumes.

•  Defini6on: A sink node in a program is a statement fragment at which program execu6on halts or terminates.

•  Defini6on: A module execu6on path (MEP) is a sequence of statements that begins with a source node and ends with a sink node, with no intervening sink nodes.

•  Defini6on: A message is a programming language mechanism by which one unit transfers control to another unit.

Extensions to Defini6ons

More defini(ons

•  Defin(on: An MM-‐Path is an interleaved sequence of module execu(on paths (MEP) and messages.

•  Defini(on: Given a set of units, their MM-‐Path graph is the directed graph in which nodes are module execu6on paths and edges correspond to messages and returns from one unit to another.

Defini6ons for Integra6on Tes6ng

1

2

3 4

5

6

A

Example (cont.): source nodes and sink nodes

Sink nodes

Source nodes


1

2

3

4

B


Sink nodes

Source nodes


1

2 3

4

5

C


Sink nodes

Source nodes



1

2

3 4

5

6

1

2

3

4

1

2 3

4

5

A B C

Sink nodes

Source nodes



•  Module A: –  Source nodes: Nodes 1 and 5 –  Sink nodes: Nodes 4 and 6

•  Module B: –  Source nodes: Nodes 1 and 3 –  Sink nodes: Nodes 2 and 4

•  Module C: –  Source nodes: Node 1 –  Sink nodes: Node 5

Example (cont.) module execu(on path (MEP)

•  MEP (A, I) = < 1, 2, 3, 6 > •  MEP (A, II) = < 1, 2, 4 > •  MEP (A, III) = < 5, 6 > •  MEP (B, I) = < 1, 2 > •  MEP (B, II) = < 3, 4 > •  MEP (C, I) = < 1, 2, 4, 5 > •  MEP (C, II) = < 1, 3, 4, 5 >

Crea(ng a MM-‐path graph Example (cont.) module execu6on path (MEP)

1

2

3 4

5

6

1

2

3

4

1

2 3

4

5

A B C

MEP(A,II)

MEP(A,I)

MEP(B,I) MEP(C,II)

MEP(C,I)

MEP(B,II)

MEP(A,III)

MM-‐path graph example Example (cont.): MM-‐Path graph

MEP (B, I)

MEP (C, I) MEP (A, I)

MEP (A, III)

MEP (B, II)

MEP (C, II)

MEP (A, II)

Messages!

Returns!

Test cases are selected to cover these paths

Problems

•  more effort is needed to iden6fy the MM-‐Paths. This effort is probably offset by the elimina(on of stub and driver development.

Software Testing: A Craftsman’s Approach, 3rd Edition System Testing

Chapter 14

System Testing


System Testing

• Threads ( = a system level test cases)• Basis concepts of requirements specification• Identifying threads• Metrics for system testing


Various Views of Threads• a scenario of normal usage• a Use Case• a system level test case• a stimulus/response pair• behavior that results from a sequence of

system level inputs• an interleaved sequence of port input and

output events• a sequence of transitions in a state machine

description of the system• an interleaved sequence of object messages

and method executions• a sequence of machine instructions• a sequence of source instructions• a sequence of atomic system functions


Some Observations

• Threads are “dynamic”• Threads occur at execution time• Threads can be identified in (or even better:

derived from) many models– Finite state machines– Decision tables– Statecharts– Petri nets– Use Cases (sufficiently detailed)


Candidate Threads in the SimpleATM System

• Entry of a digit• Entry of a Personal Identification Number (PIN)• A simple transaction: ATM Card Entry, PIN

entry, select transaction type (deposit,withdraw), present account details (checking orsavings, amount), conduct the operation, andreport the results.

• An ATM session, containing two or moresimple transactions.


Levels of Threads

• A unit level thread is a path in the programgraph of a unit.

• An integration level thread is an MM-Path.• There are two levels of system level threads:– A single thread– A set of interacting threads

• If necessary, we can deal with threads insystems of systems


System Level Threads

• An atomic system function (ASF) is an action that isobservable at the system level in terms of port input andoutput events.

• Given a system defined in terms of atomic systemfunctions, the ASF Graph of the system is the directedgraph in which nodes are atomic system functions andedges represent sequential flow.

• A source ASF is an atomic system function that appearsas a source node in the ASF graph of a system; similarly,a sink ASF is an atomic system function that appears asa sink node in the ASF graph.

• A system thread is a path from a source ASF to a sinkASF in the ASF graph of a system.


ASFs and MM-Paths

• MM-Paths “end” at a point of messagequiescence.

• In an event-driven system, ASFs frequently occur“between” points of event quiescence.

• There is no nice connection between ASFs andMM-Paths.

• An ASF corresponds to a “stimulus/responsepair”.– “stimulus/response cluster” is more accurate.– depending on its context, an input event may result in

several distinct output events.


Classifying the Candidate Threads

• an MM-Path: Entry of a digit• an ASF: Entry of a Personal Identification

Number (PIN)• a thread: A simple transaction: ATM Card

Entry, PIN entry, select transactiontype (deposit, withdraw), presentaccount details (checking or savings, amount), conduct the operation, and report the results.

• a sequence An ATM session, containing twoof threads: or more simple transactions.


Closer Look at the PIN Entry ASF

• a sequence of system level inputs and outputs:– A screen requesting PIN digits– An interleaved sequence of digit keystrokes and

screen responses– The possibility of cancellation by the customer before

the full PIN is entered– A system disposition (depending on the validity of the

PIN

• Observe:– several stimulus/response pairs– this is the cross-over point between integration and

system testing


E/R Model of Basis Concepts

Mainline requirements specification techniques populate some (or all ) portions of this database.

Data

Event

Action

Device Thread

1..n

1..n

1..n

1..n

1..n

1..n

input

output

occur

sequenceOf

1..n

1..n


Data

Event

Action

Device

Thread

Structural Model

Context Model

Behavior Model

Modeling with the Basis Concepts

Condition


1. Card Entry

2.1 First PIN

Try

2.2 Second

PIN Try

2.3 Third PIN

Try

3. Await

Transaction

Choice

DepositBalance Withdrawal

Print Receipt

Incorrect PIN0.10

Incorrect PIN

0.10

Legitimate card 0.95

Correct PIN 0.90

Correct PIN 0.90

Correct PIN 0.90

Incorrect PIN

0.10

Button B1

0.05 Button B2

0.10

Button B3

0.85

1.001.00 Normal 0.85

Low cash 0.10

Low balance 0.05

Bad card 0.05


Idle

Awaiting First

PIN Try

Await Transaction

Choice

Awaiting Second PIN Try

Wrong Card Display Screen S1,

Eject Card

/ Display Screen S1

Legitimate Card Display Screen S2

Correct PINDisplay Screen S5

Incorrect PIN Display Screen S3

Awaiting Third

PIN Try


Correct PIN Display Screen S5

Correct PIN Display Screen S5


Decomposition of Await PIN State(PIN Entry FSM)


Correct Pin

2.x.10 Digits

Received

2.x.21 Digit

Received

2.x.32 Digits

Received

2.x.43 Digits

Received

2.x.6Cancel

Hit

digit / echo 'X---'

digit / echo 'XX--'

digit / echo 'XXX-'

digit / echo 'XXXX'

cancel

2.x.54 Digits

Received

cancel

cancel

cancel

Cancelled

Incorrect Pin

x1

x2

x3

x4

x5 x6

x7

x8

x9

x10 x11

Port Input Events! digit!! cancel!

Port Output Events! echo 'X---'! echo 'XX--'! echo 'XXX-'! echo 'XXXX'

Logical Output Events! Correct PIN! Incorrect PIN! Canceled

PIN Try finite state machine(PIN Try x)


Deriving An ASF Test Case

Description:! Correct PIN on First Try

Port Event Sequence in PIN Try FSM

Port Input Event!! ! Port Output Event

! ! ! ! ! ! Screen 2 displayed with '----'1 pressed!! ! ! ! ! ! Screen 2 displayed with 'X---'2 pressed!! ! ! ! ! ! Screen 2 displayed with 'XX--'3 pressed!! ! ! ! ! ! Screen 2 displayed with 'XXX-'4 pressed!! ! ! ! ! ! Screen 2 displayed with 'XXXX'

(Correct PIN)! ! ! Screen 5 displayed


Deriving An ASF Test Case (cont'd)

Description:! Correct PIN on First Try

Test operator instructions

Initial Conditions: screen 2 being displayed with no digit echoes

Perform the following sequence of steps:1.! Verify:! ! Screen 2 displayed with '----'2.! Cause:! ! 1 pressed!3.! Verify:! ! Screen 2 displayed with 'X---'4.! Cause:! ! 2 pressed!5.! Verify:! ! Screen 2 displayed with 'XX--'6.! Cause:! ! 3 pressed!7.! Verify:! ! Screen 2 displayed with 'XXX-'8.! Cause:! ! 4 pressed!9.! Verify:! ! Screen 2 displayed with 'XXXX'

Post Condition:! Screen 5 displayed

Test Result:!___ Pass! ! ! ___ Fail


Slight Digression: Architecture of an Automated Test Executor

ATEProcessor Harness

PortBoundary

Cause Digit Keypress

Digit Keypress

Verifyscreen

text

screentext


Metrics for System Testing

In the PIN Entry ASF, for a given PIN, there are 156 distinct paths from the First PIN Try state to the Await Transaction Choice or Card Entry states in the PIN Entry FSM. Of these, 31 correspond to eventually correct PIN entries (1 on the first try, 5 on the second try, and 25 on the third try); the other 125 paths correspond to those with incorrect digits or with cancel keystrokes.

To control this explosion, we have two possibilities:

•! pseudo-structural coverage metrics•! "true" structural coverage metrics


Pseudo-structural Coverage Metrics

Behavioral models provide "natural" metrics:

Decision Table Metrics:! •! every condition! ! ! ! ! ! ! ! •! every action! ! ! ! ! ! ! ! •! every rule

FSM Metrics:! ! ! ! •! every state! ! ! ! ! ! ! ! •! every transition

Petri Net Metrics:! ! ! •! every place! ! ! ! ! ! ! ! •! every port event! ! ! ! ! ! ! ! •! every transition! ! ! ! ! ! ! ! •! every marking

These are pseudo-structural because they are just models of the eventual system.


Pseudo-structural Coverage Metrics for PIN Try

Input Event! ! State Sequence! ! Sequence1,2,3,4! ! ! 2.x.1, 2.x.2, 2.x.3, 2.x.4, 2.x.51,2,3,5! ! ! 2.x.1, 2.x.2, 2.x.3, 2.x.4, 2.x.5C! ! ! ! 2.x.1, 2.x.61,C! ! ! ! 2.x.1, 2.x.2, 2.x.61,2,C! ! ! 2.x.1, 2.x.2, 2.x.3, 2.x.61,2,3,C! ! ! 2.x.1, 2.x.2, 2.x.3, 2.x.4, 2.x.6

Two test cases yield state coverage

Input Event! ! Path of Sequence! ! Transitions

1,2,3,4! ! ! x1, x2, x3, x4, x51,2,3,5! ! ! x1, x2, x3, x4, x6C! ! ! ! x7, x111,C! ! ! ! x1, x8, x111,2,C! ! ! x1, x2, x9, x111,2,3,C! ! ! x1, x2, x3, x10, x11

All six are needed for transition coverage


Pseudo-structural Coverage Metrics for PIN Entry FSM

Input Event! ! State Sequence! ! Sequence

1,2,3,4! ! !

1,2,3,5,1,2,3,4! ! !

1,2,3,5,C,1,2,3,4

C,C,C! ! ! !

! ! !

How many test cases yield state coverage?

Input Event! ! Path of Sequence! ! Transitions

1,2,3,4! ! !

1,2,3,5,1,2,3,4! ! !

1,2,3,5,C,1,2,3,4

C,C,C! !

How many are needed for transition coverage?


Consequences of Pseudo-structural Coverage Metrics

1.! Combinatoric explosion is controlled! Selecting test cases from the FSM decomposition reduced

! 156 threads to 10 test cases

2.! Fault isolation is improved! When a "verify" operation fails, use the FSMs to determine:! •! what went wrong! •! where it went wrong! •! when it went wrong

3.! Base information for testing management


SATM System Threads

1.! Insertion of an invalid card. (this is probably the "shortest" system thread)

2.! Insertion of a valid card, followed by three failed PIN entry attempts.

3.! Insertion of a valid card, a correct PIN entry attempt, followed by a balance inquiry.

4.! Insertion of a valid card, a correct PIN entry attempt, followed by a deposit.

5.! Insertion of a valid card, a correct PIN entry attempt, followed by a withdrawal.

6.! Insertion of a valid card, a correct PIN entry attempt, followed by an attempt to withdraw more cash than the account balance.


SATM System Thread Testing

SATM Test Data

ATM ! PAN! Expected! Checking! SavingsCard! ! ! PIN!! ! Balance!! Balance1! ! 100!! 1234! ! $1000.00! $800.002! ! 200!! 4567! ! $100.00! ! $90.003! ! 300!! 6789! ! $25.00! ! $20.004! ! (invalid)

Port Input Events! ! ! ! Port Output Events

•! Insert ATM Card (n)! ! ! •! Display Screen(n,text)•! Key Press Digit (n)! ! ! •! Open Door(dep, withdraw)•! Key Press Cancel! ! ! •! Close Door(dep, withdraw)•! Key Press Button B(n)! ! •! Dispense Notes (n)•! Insert Deposit Envelope! ! •! Print Receipt (text)! ! ! ! ! ! ! ! •! Eject ATM Card


Thread 1 Test Procedure

Description: invalid card Test operator instructions Initial Conditions: screen 1 being displayed Perform the following sequence of steps: 1. Cause: Insert ATM Card 4 2. Verify: Eject ATM Card 3. Verify: Display Screen(1, null) Post Condition: Screen 1 displayed Test Result: ___ Pass ___ Fail


Thread 2 Test ProcedureDescription: valid card, 3 failed PIN attempts

Initial Conditions: screen 1 being displayedPerform the following sequence of steps:

1. Cause: Insert ATM Card 12. Verify: Display Screen(2, '----')1. Cause: Key Press Digit (1)2. Verify: Display Screen(2,'X---')1. Cause: Key Press Cancel2. Verify: Display Screen(2,'----')1. Cause: Key Press Digit (1)2. Verify: Display Screen(2,'X---')1. Cause: Key Press Digit (2)2. Verify: Display Screen(2,'XX--')1. Cause: Key PressCancel2. Verify: Display Screen(2,'----')1. Cause: Key Press Digit (1)2. Verify: Display Screen(2,'X---')1. Cause: Key Press Digit (2)2. Verify: Display Screen(2,'XX--'1. Cause: Key Press Digit (3)2. Verify: Display Screen(2,'XXX-')1. Cause: Key Press Digit (5)2. Verify: Display Screen(2,'XXXX')3. Verify: Display Screen(4, null)3. Verify: Display Screen(1, null)

Post Condition: Screen 1 displayed Test Result: ___ Pass___ Fail


Thread 3 Test ProcedureDescription: valid card, a correct PIN entry attempt, followed by a balance inquiry of the checking account. Initial Conditions: screen 1 being displayed Perform the following sequence of steps: 1. Cause: Insert ATM Card 1 2. Verify: Display Screen(2, '----') 3. Cause: Key Press Digit (1) 4. Verify: Display Screen(2,'X---') 5. Cause: Key Press Digit (2) 6. Verify: Display Screen(2,'XX--' 7. Cause: Key Press Digit (3) 8. Verify: Display Screen(2,'XXX-') 9. Cause: Key Press Digit (4) 10. Verify: Display Screen(2,'XXXX') 11. Verify: Display Screen(5, null) 12. Cause: Key Press Button B(1) 13. Verify: Display Screen(6, null) 14. Cause: Key Press Button B(1) 15. Verify: Display Screen(14,null) 16. Cause: Key Press Button B(2) 17. Verify: Print Receipt ('$1000.00') 18. Verify: Display Screen(15, null) 19. Verify: Eject ATM Card 20. Verify: Display Screen(1, null) Post Condition: Screen 1 displayed Test Result: ___ Pass ___ Fail


Thread 4 Test ProcedureDescription: valid card, a correct PIN entry attempt, followed by a $25.00 deposit to the checking account 1. Cause: Insert ATM Card 1 2. Verify: Display Screen(2, '----') 3,5,7,9. Cause: Key Press Digit (1,2,3,4) 4,6,8,10. Verify: Display Screen(2,'XXXX') 11. Verify: Display Screen(5, null) 12. Cause: Key Press Button B(2) 13. Verify: Display Screen(6, null) 14. Cause: Key Press Button B(1) 15. Verify: Display Screen(7, '$----.--') 16. Cause: Key Press Digit (2) 17. Verify: Display Screen(7, '$----.-2') 18. Cause: Key Press Digit (5) 19. Verify: Display Screen(7, '$----.25') 20. Cause: Key Press Digit (0) 21. Verify: Display Screen(7, '$---2.50') 22. Cause: Key Press Digit (0) 23. Verify: Display Screen(7, '$--25.00' 24. Verify: Display Screen(13, null) 25. Verify: Open Door(deposit) 26. Cause: Insert Deposit Envelope 27. Verify: Close Door(deposit) 28. Verify: Display Screen(14, null) 29. Cause: Key Press Button B(2) 30. Verify: Print Receipt ('$1025.00') 31. Verify: Display Screen(15, null) 32. Verify: Eject ATM Card 33. Verify: Display Screen(1, null) Post Condition: Screen 1 displayed Test Result: ___ Pass ___ Fail


Thread 5 Test Procedure

Description: valid card, a correct PIN entry attempt, followed by an attempt to withdraw more cash than the savings account balance. 1. Cause: Insert ATM Card 2 2. Verify: Display Screen(2, '----') 3,5,7,9. Cause: Key Press Digit (4,5,6,7) 4,6,8,10. Verify: Display Screen(2,'XXXX') 11. Verify: Display Screen(5, null) 12. Cause: Key Press Button B(3) 13. Verify: Display Screen(6, null) 14. Cause: Key Press Button B(2) 15. Verify: Display Screen(7, '$----.--') 16,18,20,22,24 Cause: Key Press Digit (1,1,0,0,0) 17,19,21,23,25 Verify: Display Screen(7, '$-110.00') 24. Verify: Display Screen(8, '----.--') 26. Cause: Key Press Cancel 28. Verify: Display Screen(14, null) 29. Cause: Key Press Button B(2) 30. Verify: Print Receipt ('$90.00') 31. Verify: Eject ATM Card 32. Verify: Display Screen(1, null) Post Condition: Screen 1 displayed Test Result: ___ Pass ___ Fail


Operational Profiles

Zipf's Law: 80% of the activities occur in 20% of the space

•! productions of a language syntax•! natural language vocabulary•! menu options of a commercial software package•! area of an office desktop•! floating point divide on the Pentium chip

For Threads: a small fraction of all possible threads represents the majority of system execution time.

Therefore: find the occurrence probabilities of threads and use these to order thread testing.


1. Card Entry

2.1 First PIN

Try

2.2 Second

PIN Try

2.3 Third PIN

Try

3. Await

Transaction

Choice

DepositBalance Withdrawal

Print Receipt

Incorrect PIN0.10

Incorrect PIN

0.10

Legitimate card 0.95

Correct PIN 0.90

Correct PIN 0.90

Correct PIN 0.90

Incorrect PIN

0.10

Button B1

0.05 Button B2

0.10

Button B3

0.85

1.001.00 Normal 0.85

Low cash 0.10

Low balance 0.05

Bad card 0.05

Common Thread Probabilities Legitimate Card 0.95 PIN ok 1st try 0.90 Withdraw 0.85 Normal 0.85 0.6177375 Rare Thread Probabilities Legitimate Card 0.95 Bad PIN 1st try 0.10 Bad PIN 2nd try 0.10 PIN ok 3rd try 0.90 Withdraw 0.85 Low Cash 0.01 0.00072675

Operational Profiles of SATM


SATM Atomic System Functions

ASF1: Examine ATM Card Inputs: PAN from card, List of acceptable cards Outputs: Legitimate Card, Wrong Card

ASF2: Control PIN Entry Inputs: Expected PIN, Offered PIN Outputs: PIN OK, Wrong PIN

ASF3: Get Transaction Type Inputs: Button1, Button2, or Button3 depressed Outputs: call Get Account Type (not externally visible)

ASF4: Get Account Type Inputs: Button1 or Button2 depressed Outputs: call one of Process Withdrawal, Process Deposit, or Display Balance (not externally visible)

ASF5: Process Withdrawal Inputs: Amount Requested, Cash Available, Local Balance Outputs: Process Request (call Dispense Cash) Reject Request (insuficient funds or insuficient balance)


SATM Atomic System Functions

ASF6: Process Deposit Inputs: Deposit Amount , Deposit Envelope, Deposit Door Status, Local Balance Outputs: Process Request (call Credit Local Balance) Reject Request

ASF7: Display Balance Inputs: Local Balance Outputs: (call Screen Handler) ASF8: Manage Session Inputs: New Transaction Requested, Done Outputs: (call Get Transaction Type or call Print Receipt)

ASF9: Print Receipt Inputs: Account Number, transaction type and amount, new local balance, time, date Outputs: formatted text for receipt, (call Eject Card)

ASF10: Eject Card Inputs: (invoked) Outputs: (control rollers)


"Hidden" Atomic System Functions

ASF11: Dispense $10 Note

ASF12: Screen Handler

ASF13: Button Sensor

ASF14: Digit Keypad Sensor

ASF15: Cancel Sensor

ASF16: Card Roller Controller

ASF17: Control Deposit Door

ASF18: Control Deposit Rollers

ASF19: Control Cash Door

ASF20: Count Cash on Hand

ASF21: Timer


SATM Threads

Thread 1: Wrong Card State Sequence: Idle, Idle Event Sequence: Display Screen 1, Wrong Card ASF Sequence:!

Thread 2: Wrong PIN State Sequence: Idle, Await 1st PIN Try, Await 2nd PIN Try, Await 3rd PIN Try, Idle Event Sequence: Display Screen 1, Legitimate Card, Wrong PIN, Wrong PIN, Wrong PIN, Display Screen 4, Display Screen 1 ASF Sequence:!

Thread 3: Balance Inquiry State Sequence: Idle, Await 1st PIN Try, Acquire Transaction Data, Display Balance, Display Balance, Close Session, Idle Event Sequence: Display Screen 1, Legitimate Card, Wrong PIN, Wrong PIN, Wrong PIN, Display Screen 4, Display Screen 1 ASF Sequence:!


Thread 4: Balance then Deposit then Withdraw State Sequence: Idle, Await 1st PIN Try, Acquire Transaction Data, Display Balance, Close Session, Acquire Transaction Data, Process Deposit, Close Session, Acquire Transaction Data, Process Withdrawal, Close Session, Idle Event Sequence: Display Screen 1, Legitimate Card, Wrong PIN, Wrong PIN, Wrong PIN, Display Screen 4, Display Screen 1 ASF Sequence:!

Thread 5: 3 PIN Tries then Balance then Deposit then Withdraw State Sequence: Idle, Await 1st PIN Try, Await 2nd PIN Try, Await 3rd PIN Try, Acquire Transaction Data, Display Balance, Close Session, Acquire Transaction Data, Process Deposit, Close Session, Acquire Transaction Data, Process Withdrawal, Close Session, Idle Event Sequence: Display Screen 1, Legitimate Card, Wrong PIN, Wrong PIN, Wrong PIN, Display Screen 4, Display Screen 1 ASF Sequence:!

SATM Threads


ATM States and Transitions

S1

S2 S3 S4

S5

S6 S7 S8

S9

t1

t2

t3 t4

t5

t6 t7 t8

t9t10 t11

t12t13 t14

t15

t16

Software Testing: A Craftsman’s Approach, 3rd Edition Interaction Testing

Chapter 15

Interaction Testing


Interaction: the hot topic of the 1990s

• "Elements of Interaction"Communications of the ACM, (Jan. 93)Robin Milner (1993 Turing Award Lecture)

• "An Investigation of the Therac-25 Accident“, IEEE Computer,(July 93) Nancy G. Leveson and Clark S. Turner.

• "Feature Interactions and Formal Specifications inTelecommunications" IEEE Computer, (Aug. 93) Pamela Zave

• The underlying issue: Non-Determinism• Our Approach:

– A Taxonomy of Interactions– An executable specification– Analysis from graph theory



Definitions of Interaction

1. "System behavior as a whole does not satisfy the separate specifications of all its [parts]." (Pamela Zave) 2. Relationship between the whole and its parts. 3. Totality of connections among components. 4. Consequences of connections among components. 5. The result of composition. (Robin Milner)


Feature InteractionFeature: a service provided by a system (activated by a subscriber paying for the service) Telephony Examples (from P. Zave): Notation: d1, d2, d3, and d4 are directory numbers • POTS (Plain Old Telephone Service) • Call Forwarding: calls to d1 are terminated on d2 iff call forwarding is enabled on d1 and activated by defining d2 as the current destination • Calling Party Identification: when active on d2, d2 receives the directory number of all incoming calls • Call Rejection: allows a subscriber to define a list of directory numbers from which calls will not be completed. • Busy Treatment: several possibilities; call override, call rejection, call forward, call forward on busy, do not disturb, automatic re-call


(Zaveʼs) Sample Interactions1. Calling Party Identification and Call Rejection calling party identification offers a directory number (data) which is used as an input in the call rejection process. 2. Call Forwarding and Call Rejection d2 rejects calls from d1, d1 forwards calls to d2, d3 calls d1. 3. Call Forward Loop d1 forwards calls to d2 d2 forwards calls to d3 d3 forwards calls to d1 d4 calls d1 4. Voice Mail and Credit Card Calling • In credit card calling, # terminates a call so that a new call can be dialed without re-entering the credit card digits. • In many Voice Mail systems, # is a command (e.g., to hear your recorded message) • d1 makes a credit card call to d2 and gets d2's voice mail. What happens when d1 enters # ?


First Clues• Feature interaction is a consequence of adaptive maintenance (i.e., adding new capabilities to an existing system) • Interactions involve connections, and the essence of every modeling technique is to find/establish connections (see Wurmann, Information Anxiety ) • Composition creates connections. What can be connected? • First Approximation:

interaction Should be Should not be

Is

Is not

intended unintended

missing null


Definitions of Determinism

Let C be some calculation, C (input) = output .

Definition 1:! If the result of C can always be predicted, C is deterministic! ! ! (or pre-determined), otherwise C is non-deterministic.

Definition 2:! If the result of C is always the same, C is deterministic! ! ! (or pre-determined), otherwise C is non-deterministic.

Example: ComputeSalesTax(price, taxrate)

If the tax rate is 4%, sales tax on a $100 item will be $4.00.

If the legislature changes the tax rate to 6%, we have interaction.

If we understand all the points of interaction, the function still is deterministic.

"Concurrency inflicts non-determinism." Robin Milner


Toward a Taxonomy of InteractionsWhat elements can interact? In what ways can these elements interact? "Basis" system elements (the reality of any system):

Data

Event

Action

Device Thread


Basis Concepts for Requirements Specification1! DataWhen a system is described in terms of its data, the focus is on the information used and created by the system. Data refers to information that is either initialized, stored, updated, or (possibly)

destroyed.

2! ActionsActions have inputs and outputs, and these can be either data or port events. Some methodology-specific synonyms for actions: transform, data transform, control transform, process, activity, task, method, and service. Actions can be decomposed into lower level actions.

3! DevicesEvery system has devices; these are the sources and destinations of system level inputs and outputs (events that occur at the port boundary). Physical actions (e.g., keystrokes, light emissions from a screen) occur on port devices, and these are translated from physical to logical (or logical to physical)

appearances by actions that execute on other devices (e.g., a CPU executing software).

4! EventsA system level input (or output) that occurs on a port device. Like data, events can be inputs to or outputs of actions. Events can be discrete (such as keystrokes) or they can be continuous (such as temperature, altitude, or pressure). There are situations where the context of present data values changes the logical meaning of physical events. We refer to such situations as "context sensitive port events".

5! ThreadsA thread an instance of execution-time behavior of a system. Two synonyms: a scenario, a use-case. A thread is a sequence of actions, and these in turn have data and events as their inputs and outputs.


Connections (interactions) Among Basis Concepts

Data

Action

Event

Thread

Data Action Device Event Thread

Square ofOpposition

I/O,usage

contextsensitivity

points ofn-connection

resourcecontention

incidence

timing, raceconditions

timing,concurrency

n-connectivity


I/O,usage

contextsensitivity



incidence

Device

execution

execution

I/O,usage

I/O,usage


Toward a Taxonomy of Interactions

•! Each element needs another "dimension"; we'll call it Location, where location refers to time and position.

•! Each of these elements can interact with itself.

•! Interactions among pairs of elements are more interesting.

Data

Event

Action

Device Thread


Some Ground RulesViews of location: • a point in space-time • something that happens in a processor (position) 1. For now, a processor is something that executes threads, or a device where events occur. 2. Since threads execute, they have a strictly positive time duration. 3. In a single processor, two threads cannot execute simultaneously. 4. Events have a strictly positive time duration. 5. Two (or more) input events can occur simultaneously, but an event cannot occur simultaneously in two (or more) processors. 6. In a single processor, two output events cannot begin simultaneously.


Taxonomy of InteractionsStatic interaction: independent of time Dynamic interaction: time dependent Each of the five "basis elements" can interact in each quadrant:

static dynamic

single processor

multiple processor

We will consider static interactions of data with data, anddynamic interactions of threads with threads.


Static Interactions (Known to Aristotle)

Interactions in the Square of Opposition. 1. Contradictories: exactly one is true 2. Contraries: cannot both be true 3. Sub-contraries: cannot both be false 4. Subalternation: Truth of superaltern guarantees truth of its subaltern Examples 1. When the pre-condition for a thread is a conjunction of data propositions, contrary or contradictory data values will prevent thread execution. 2. Context sensitive port input events usually involve contradictory data. 3. Case statement clauses are contradictories. 4. Rules in a decision table are contradictories.

All S is P No S is P

Some S is P Some S is not P

contraries

contradictories

sub-contraries

subalternation subalternation


Static Interactions in Multiple Processors

The "Call Forwarding Loop" in telephony.

Background: when a subscriber defines a call forwarding destination, this becomes call ro¨ting data local to the subscribers telephone office.

Suppose Subscriber A (in Grand Rapids) forwards calls to Subscriber B in Northbrook, and

Subscriber B (in Northbrook) forwards calls to Subscriber C in Phoenix, and

Subscriber C (in Phoenix) forwards calls to Subscriber A inGrand Rapids.

What happens when someone outside this loop calls one of A, B, or C?

Observations:

1.! The call forwarding data is locally correct, but globally, it is contrary.

2.! The global contrary condition is afault, not a failure.

3.! The fault only becomes a failure when a thread creates a dynamic interaction.

4.! (Most telephone systems avoid this by refusing to forward a forwarded call.)


n-Connectedness

Linear Graph Theory sheds some light on interaction.

0-connected

ji

1-connected

ji

ji

2-connected

j

i

3-connected

j

i

Two kinds of faults:

Missing n-connectedness occurs when a pair lacks an essential connection, andInappropriate n-connctedness, when a pair has an undesired connection.


Interpretations of n-Connectedness

0-Connected: true independence

1-Connected: (by an ancestor) resource conflict, context sensitivity (by a descendant) ambiguous cause

2-Connected: define-reference, enable, disable, precedence, prohibit prerequisite

3-Connected: mutual influence, repetition, deadlock

Examples:

Failures under the "single failure" assumption of Reliability Theory

Context Sensitive Port Events

Falkland Island submarine incident

Email loop


Data with Data

0-Connected! data that are independent

1-Connected! data that are inputs to the same action

2-Connected! data that are sub-alternates,! ! ! ! data that are used in a computation

3-Connected! data that are contraries, contradictories, or sub-contraries,! ! ! ! "deeply" related data, as in iteration or semaphores

Threads with Threads

•! Threads can be n-connected with each other in two ways: via events! and/or via data.•! To explore this, we need a sufficiently expressive notation.

My candidate: Event Driven Petri Nets

Interactions Based on n-Connectedness


Elements of the Interaction Taxonomy

1.! Basis concept interactions

! data-data! ! ! port-port! data-action! ! ! port-event! data-event! ! ! event-event! data-thread! ! ! event-thread! action-action! ! thread-thread! action-event2.! Refined by appropriate relationship (e.g., n-connectivity, square! of opposition)3.! Further refined by Is/Should modality

Placement of Zave's examples

1.! Calling Party Identification and Call Rejection! Intended thread-thread 2-connectivity2.! Call Forwarding and Call Rejection! Unintended data-data contraries3.! Call Forward Loop! Unintended thread-thread 3-connectivity4.! Voice Mail and Credit Card Calling! Unintended data-event context sensitivity


An Event-Driven Petri Net is a tripartite directed graph (P, D, S, In, Out) composed of three sets of nodes, P, D, and S, and two mappings, In and Out, where • P is a set of port events • D is a set of data places • S is a set of transitions • In is a set of ordered pairs from (P∪ D) × S • Out is a set of ordered pairs from S × (P ∪ D) Event-Driven Petri Nets express four of our five basic system constructs; only devices are missing. In an Event-Driven Petri Net , the external inputs are the places with indegree = 0, and the external outputs are the places with outdegree = 0. Graphical conventions for Event-Driven Petri Nets

A Petri Net Model for Interactions

port event data place

transition


Composing Event-driven Petri Nets

Given two (or more) EDPNs, their composition (unique!) is obtained as follows:

1. If any places or port events have synonymous names, collapse these into unique names. (For example, collapse Lamp ON and Light ON into Lamp ON)

2. Consider all data places and port events to be global (if a place appears in two individual nets, it need only appear once in their composition.)

3. Consider actions to be local. (This preserves the execution sequence of the individual threads.)

4. Adjust all input and output relations (arrows) to preserve the original input and output relations.

5. It may be necessary to add transitions and input/output edges to represent interactions that were not captured in the prior components.

6. Trace through some mainline threads to convince yourself that the composition (especially interactions) is accurate.


Thread Interaction and CompositionWhen two threads interact (via an event or data), we can always compose their respective EDPNs.

p1 d1

s1

p3

d3

s2

d4

p1 d2

s3

p3 d4

p1

s1

d3

s2

d1 d2

s3

p3

d4

Thread 1 Thread 2 Composition


t1

t2

t3

t4

t5

t6

p1 p2

p3

p4

p5

p6

p7

t1

t2

p1 p2

p3

t5

t6

p6

p7

p3

t3

t4

p4

p5

p3

Producer/Consumers Composition

Producer

Consumer 1 Consumer 2


The windshield wiper on the Saturn automobile (at least on the 1992 models) is controlled by a lever with a dial. The lever has four positions, OFF, INT (for intermittent), LOW, and HIGH, and the dial has three positions, numbered simply 1, 2, and 3. The dial positions indicate three intermittent speeds, and the dial position is relevant only when the lever is at the INT position. The decision table below shows the windshield wiper speeds (in wipes per minute) for the lever and dial positions.

Saturn Windshield Wiper Controller

Lever! ! OFF!! INT! INT! INT!LOW! HIGHDial!! ! n/a! ! 1! 2! 3! n/a! ! n/a

Wiper! ! 0! ! 4! 6! 12! 30! ! 60

Input Event! Description! ie1! ! lever from OFF to INT! ie2! ! lever from INT to LOW! ie3! ! lever from LOW to HIGH! ie4! ! lever from HIGH to LOW! ie5! ! lever from LOW to INT! ie6! ! lever from INT to OFF! ie7! ! dial from 1 to 2! ie8! ! dial from 2 to 3! ie9! ! dial from 3 to 2! ie10! ! dial from 2 to 1

Output Event! Description! oe1! ! ! 0 w.p.m.! oe2! ! ! 4 w.p.m.! oe3! ! ! 6 w.p.m.! oe4! ! ! 12 w.p.m.! oe5! ! ! 30 w.p.m.! oe6! ! ! 60 w.p.m.


OFF

HIGH

LOW

INT

ie1 ?

ie2oe5

ie3oe6

ie4oe5

ie5?

ie6oe1

Lever

1

3

2

ie7?

ie8?

ie10?

ie9?

Dial

Saturn Windshield Wiper FSMs

Input Event! Description! ie1! ! lever from OFF to INT! ie2! ! lever from INT to LOW! ie3! ! lever from LOW to HIGH! ie4! ! lever from HIGH to LOW! ie5! ! lever from LOW to INT! ie6! ! lever from INT to OFF! ie7! ! dial from 1 to 2! ie8! ! dial from 2 to 3! ie9! ! dial from 3 to 2! ie10!! dial from 2 to 1

Output Event! Description! oe1!! ! 0 w.p.m.! oe2!! ! 4 w.p.m.! oe3!! ! 6 w.p.m.! oe4!! ! 12 w.p.m.! oe5!! ! 30 w.p.m.! oe6!! ! 60 w.p.m.

Note that several transition actions are indeterminate because of composition.


p1

d3

d2

d1

s1

s2 s5

s6

p6

p2 p5

p15

d4

s3 s4

p3 p4

p16 p15

p11

OFF

HIGH

LOW

INT

ie1 ?

ie2 oe5

ie3 oe6 ie4

oe5

ie5 ?

ie6 oe1

Lever Input Event Description p1 ie1 lever from OFF to INT p2 ie2 lever from INT to LOW p3 ie3 lever from LOW to HIGH p4 ie4 lever from HIGH to LOW p5 ie5 lever from LOW to INT p6 ie6 lever from INT to OFF p7 ie7 dial from 1 to 2 p8 ie8 dial from 2 to 3 p9 ie9 dial from 3 to 2 p10 ie10 dial from 2 to 1 Output Event Description p11 oe1 0 w.p.m. p12 oe2 4 w.p.m. p13 oe3 6 w.p.m. p14 oe4 12 w.p.m. p15 oe5 30 w.p.m. p16 oe6 60 w.p.m. Data Place Description d1 Lever at OFF position d2 Lever at INT position d3 Lever at LOW position d4 Lever at HIGH position d5 Dial at position 1 d6 Dial at position 2 d7 Dial at position 3

Deriving an EDPN from a State Machine


p1

s1

d3

s2

d1

p3

d4

d2

s3

Let T1 and T2 be two EDPN threads in which synonym places have been resolved, and with external input and output sets EI1, EI2, EO1, and EO2. Furthermore, let T be the composition of threads T1 and T2, where EI = EI1∩ EI2 and EO = EO1 ∩ EO2 are the external input and output sets of the composed thread T. The threads T1 and T2 are: •0-connected if EI1 ∩ EI2 = ∅, EO1 ∩ EO2 = ∅, EI1 ∩ EO2 = ∅, and EO1 ∩ EI2 = ∅ •1-connected if either EI ≠ ∅ or EO ≠ ∅, •2-connected if either EI1 ∩ EO2 ≠ ∅ or EI2 ∩ EO1 ≠ ∅, •3-connected if both EI1 ∩ EO2 ≠ ∅ and EI2 ∩ EO1 ≠ ∅

n-Connected Thread Interaction


p1

d3

d2

d1

s1

s2 s5

s6

p6

p2

p15

d4

s3 s4

p3 p4

p16 p5

p11p7

s7

s8 s9

s10

p10

p8 p9

p12

p5

d6

d5

d7

p13

p14

s11

s12

s13

Full EDPN for the Saturn Windshield Wiper


Composition in a Database(all relations are 0..n at each end)

Data

Event

Device

Action Thread

DataInput

DataOutput

EventInput

EventOutput

OccursOn

SequenceOf


DataInput Table for d3

LowToInts5Lowd3LowtoHighs3Lowd3ActionNameActionIDDataNameDataID


DataInput Table for d2

Intermittentd2

Intermittentd2

Intermittentd2

Intermittentd2

Intermittentd2

ActionNameActionIDDataNameDataID


What Connections Can you Identify Between anEDPN Data Model and Graph Theory?

• Indegree of a place

• Outdegree of a place

• Indegree of a transition

• Outdegree of a transition

• Indegree of an event

• Outdegree of an event

Software Testing: A Craftsman’s Approach, 3rd Edition Object-Oriented Testing

Chapters 16 - 20

Testing Object-Oriented Software


Object-Oriented Testing

1. Traditional vs. Object-Oriented Testing 2. Saturn Windshield Wiper Example 3. Testing with O-O Notations


Traditional vs. Object-Oriented Testing

Data

Event

Action

Device Thread

input

output

Data

Event

Action

Device

Thread

encapsulate

Object

Object-orientation repackages the basis concepts (all relations are 0..n)


Issues in Object-Oriented Testing

1. Implications of message-based communication With an procedural language, program graphs are "natural". O-O testing must deal with event and message quiescence. 2. Decomposition to composition. Functional decomposition tree as a basis of integration testing is lost. Composition implies unknowable contexts. We can never know all the possible objects with which a given object may be composed. 3. O-O language packaging requires redefinition of testing levels.

What is a unit? Can we continue with integration level

constructs (MM-Path, ASF)? What do O-O threads look like?


Levels of Object-Oriented Testing

Level Unit Integration System

Item Method of an object? Class? MM-Path Atomic System Function Thread Thread Interaction

Boundary Program graph Message quiescence Event quiescence Source to sink ASF (none)

Notice the cascading levels of interaction: • unit testing covers statement interaction, • MM-Path testing covers method interaction, • ASF testing covers MM-Path interaction, • thread testing covers object interaction, and all of this culminates in • thread interaction.


Re-usable Testing Techniques

Level Unit Integration System

Item Method of an object Class MM-Path Atomic System Function Thread

Thread Interaction

Technique Traditional functional and/or structural StateChart-based New definition? New definition? New definition? (StateCharts) (as before)


The windshield wiper on the Saturn automobiles is controlled by a lever with a dial. The lever has four positions, OFF, INT (for intermittent), LOW, and HIGH, and the dial has three positions, numbered simply 1, 2, and 3. The dial positions indicate three intermittent speeds, and the dial position is relevant only when the lever is at the INT position. The decision table below shows the windshield wiper speeds (in wipes per minute) for the lever and dial positions.

Saturn Windshield Wiper Controller

Lever OFF INT INT INT LOW HIGH Dial n/a 1 2 3 n/a n/a Wiper 0 4 6 12 30 60

Input Event Description ie1 lever from OFF to INT ie2 lever from INT to LOW ie3 lever from LOW to HIGH ie4 lever from HIGH to LOW ie5 lever from LOW to INT ie6 lever from INT to OFF ie7 dial from 1 to 2 ie8 dial from 2 to 3 ie9 dial from 3 to 2 ie10 dial from 2 to 1

Output Event Description oe1 0 w.p.m. oe2 4 w.p.m. oe3 6 w.p.m. oe4 12 w.p.m. oe5 30 w.p.m. oe6 60 w.p.m.


Wiper

Speed Lever Position Dial Position

Compute Speed

Lever

position

Sense Up Sense Down

Dial

Sense Left Sense Right

position

Saturn Windshield Wiper Objects


OFF

HIGH

LOW

INT

ie1 ?

ie2 oe5

ie3 oe6 ie4

oe5

ie5 ?

ie6 oe1

Lever

1

3

2

ie7 ?

ie8 ?

ie10 ?

ie9 ?

Dial

Saturn Windshield Wiper Object FSMs

Input Event Description ie1 lever from OFF to INT ie2 lever from INT to LOW ie3 lever from LOW to HIGH ie4 lever from HIGH to LOW ie5 lever from LOW to INT ie6 lever from INT to OFF ie7 dial from 1 to 2 ie8 dial from 2 to 3 ie9 dial from 3 to 2 ie10 dial from 2 to 1

Output Event Description oe1 0 w.p.m. oe2 4 w.p.m. oe3 6 w.p.m. oe4 12 w.p.m. oe5 30 w.p.m. oe6 60 w.p.m.

Note that several transition actions areindeterminate because of composition.


OFF

HIGH

LOW

INT

ie1 m1

ie2 m2

ie3 m3 ie4

m4

ie5 m5

ie6 m6

Lever

1

3

2

ie7 m7

ie8 m8

ie10 m10

ie9 m9

Dial

0 wpm

60 wpm

30 wpm

4 wpm

Wiper

6 wpm 12 wpm

m1m1

m1

m6 m6 m6

m2 m2 m2

m5m5

m5

m3m6

m7 m8

m9m10

Resolving the Indeterminacy with Messages

Messages m1 to m6 inform the Wiper object of the state of the Lever object. Messages m7 to m10 inform the Wiper object of the state of the Dial object.


instate(Int)

Instate(1)

60

30

12

6

4

0

instate(Int)instate(Low)

instate(Off)

instate(High)instate(Low)

Instate(2)

Instate(3)

Instate(1)

Instate(2)

Instate(2)

Instate(3)

leverUpleverDown

Off

Int

Low

High

dialUpdialDown

3

2

1

dialUp

dialDown

power on

power off

leverDown

leverDownleverUp

leverUp

Windshield Wiper StateChart


• definition in UML • two kinds of o-o software: - data-driven - event-driven • how are these described (for a tester)? • what is an o-o unit: - a class? - a method? • what is the basis for integration testing?

Testing Object-Oriented Software


First Example (data-driven)

The o-oCalendar program is an object-oriented implementation of the NextDate function. (NextDate(Mar., 5, 2002)) = Mar., 6, 2002. When this is implemented in procedural code, it is approximately 50 lines long, with a cyclomatic complexity less than 15. (can be 11.) A "pure" (i.e., good practice) object-oriented implementation contains one abstract class and five classes. The next few slides show the UML description of o-oCalendar.


CalendarUnit 'abstract class currentPos As Integer CalendarUnit( pCurrentPos) setCurrentPos( pCurrentPos) increment() 'boolean

Day Month m Day( pDay, Month pMonth) setCurrentPos( pCurrentPos) setDay( pDay, Month pMonth) getDay() increment()

Month private Year y private sizeIndex = <31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31> Month(pcur, Year pYear) setCurrentPos(pCurrentPos) setMonth(pcur, Year pYear) getMonth() getMonthSize() increment()

Year Year(int pYear) setCurrentPos( pCurrentPos) getYear() increment() isleap() 'boolean

testIt Date Day d Month m Year y Date( pDay, pMonth,, pYear) increment() printDate()

Classes


CalendarUnit

Day Month Year

Date

Day Month Year

Inheritance and Aggregation


3

2

1testIt

4

5

6

7

Date.constructor Date.increment19

20

Date.printDate

10

8

9

11

12 15

1613

14

17

18

Program Graphs of Date Methods


Program Graphs of Day Methods

25

24

23

Day.setDay

26

27

Day.getDay

29

28

30

31 32

33

Day.incrementDay.constructor

23

21

22

a

b

Day.setCurrentPos


Program Graphs of Month Methods

40

39

39

Month.constructora

b

Month.setCurrentPos

38

37

36

Month.setMonth

40

39

Month.getMonth Month.getMonthSize

48

47

49

50 51

52

Month.increment41

44

42

43

46

45


Program Graphs of Year Methods

54

53

Year.constructor

b

a

Year.setCurrentPos

55

56

Year.getYear

57

58

59

Year.increment

61

62 63

64

60

Year.isLeap


Month Date

Year

Day

testIt

1: create 2: increment 3: printDate

1: create 2: increment 3: getYear

1: create 2: increment 3:setMonth 4: getMonth

1: create 2: increment 3: setDay 4: getDay

1:getMonthSize

1:isLeap

Collaboration Diagram


testit

Main

Date

Date()

increment()

printDate

Day

setCurrentPos()

Day()

increment()

setDay

getDay

Month

setCurrentPos()

Month()

increment()

setMonth

getMonth

getMonthSize

Year

setCurrentPos()

Year()

increment()

isLeap

getYear

m1

m2

m3

m15

m16

m17

m18

m19

m20

m21

m6

m7

m11m13

m5

m8

m10

m4

m9

m14

All Message Flows


testIt

1: printDate()

2:getMonth()

3:getDay()

4:getYear()

Date:testdate Month:m Year:yDay:d

time

Message Sequence Diagram (for an easy test case)


testit

Main

Date

Date()

increment()

printDate

Day

setCurrentPos()

Day()

increment()

setDay

getDay

Month

setCurrentPos()

Month()

increment()

setMonth

getMonth

getMonthSize

Year

setCurrentPos()

Year()

increment()

isLeap

getYear

m1

m15

m16

m18

m19

m21

m6

m5

m4

Message Flow for Easy Case


Currency Converter

U. S. Dollar amount

Equivalent in …

Brazil

Canada

European Community

Japan

Clear

Quit

Compute

Second Example (event-driven)


Procedural Implementation of Compute Button

procedure Compute ( USDollarAmount, EquivCurrencyAmount)dim brazilRate, canadaRate, euroRate, japanRate, USDollarAmount As SingleIf (optionBrazil) Then EquivCurrencyAmount = brazilRate * USDollarAmount Else If (optionCanada) Then EquivCurrencyAmount = canadaRate * USDollarAmount Else If (optionEuropeanUnion) Then EquivCurrencyAmount = euroRate * USDollarAmount Else If (optionJapan) Then EquivCurrencyAmount = japanRate * USDollarAmount Else Output ("No country selected") EndIF EndIF EndIFEndIF


“Object-Oriented”

Implementationof Compute

Button

object optionBrazil () Constant USdollarToBrazilReal= 2.067 private procedure senseClick commandCompute(USdollarToBrazilReal) End senseClickobject optionCanada() Constant USdollarToCanadianDollar = 1.16 private procedure senseClick commandCompute(USdollarToCanadianDollar) End senseClickobject optionEuropeanUnion () Constant USdollarToEuro= 0.752 private procedure senseClick commandCompute(USdollarToEuro) End senseClickobject optionJapan () Constant USdollarToJapanYen = 117.82 private procedure senseClick commandCompute(USdollarToJapanYen) End senseClickprocedure comandCompute ( exchangeRate)dim exchangeRate, USDollarAmount As Single USDollarAmount = Val(txtUSDollarAmount.text) EquivCurrencyAmount = exchangeRate * USDollarAmountEnd procedure Compute


“Visual Basic” Style Implementation

Public exchangeRate As Single

Private Sub optBrazil_Click() exchangeRate = 2.067End SubPrivate Sub optCanada_Click() exchangeRate = 1.16End SubPrivate Sub optEuropeanUnion_Click() exchangeRate = 0.752End SubPrivate Sub optJapan_Click() exchangeRate = 117.82End SubPrivate Sub commandCompute () EquivCurrencyAmount = exchangeRate * Val(txtUSDollarAmount.text)End Sub


Input Events for the Currency Converter

Click on OK in errormessage

ip6Click on EuropeanCommunity

ip2.3

Click on Quit buttonip5Click on Canadaip2.2

Click on Clear buttonip4Click on Brazilip2.1

Click on Compute buttonip3Click on a country buttonip2

Click on Japanip2.4Enter US Dollar amountip1

Input EventsInput Events


Output Events for the Currency Converter

Indicate Japanop3.4Reset equivalent currency amountop10Indicate European Communityop3.3Reset US Dollar Amountop9Indicate Canadaop3.2

Error Msg: Must select a countryand enter US Dollar amount

op8Indicate Brazilop3.1

Error Msg: Must enter US Dollaramount

op7Indicate selected countryop3Error Msg: Must select a countryop6Display ellipsisop2.5Display foreign currency valueop5Display Japanese Yenop2.4

Reset Japanop4.4Display European CommunityEuros

op2.3Reset European Communityop4.3Display Canadian Dollarsop2.2Reset Canadaop4.2Display Brazilian Realsop2.1Reset Brazilop4.1Display currency nameop2Reset selected countryop4Display US Dollar Amountop1

Output EventsOutput Events


ASFs and Data Places for theCurrency Converter

Sense Click on Quit buttons8Sense Click on E. U.s4

Sense Click on Clear buttons7Sense Click on Canadas3

Sense Click on Computebutton

s6Sense Click on Brazils2

Sense Click on Japans5Store US Dollar amounts1

Atomic System FunctionsAtomic SystemFunctions

Country selectedd2US Dollar Amountentered

d1

Data PlacesData Places


Missing Country

and Dollar

Message

Missing US Dollar Message

US Dollar

Amount Entered

Missing Country Message

Both Inputs Done

Country Selected

Idle

Equiv Amount

Displayed

ip1/op1ip2/op2, op3

ip2/op2, op3ip3/op7ip3/op6

ip6

ip6

ip3/op5

ip6

ip3/op8

ip4 or ip5 op2.5, op4

op9, and op10

ip4 or ip5

ip1/op1

High Level Finite State Machine


Canada

Euro Comm

Japan

Brazil

Idle

ip2.2 op2.2 op3.2 op4.1

ip2.1 op2.1 op3.1 op4.2

ip2.3 op2.3 op3.3 op4.4

ip2.4 op2.4 op3.4 op4.3

ip2.4 op2.4 op3.4 op4.2

ip2.2 op2.2 op3.2 op4.4

ip2.1 op2.1 op3.1 op4.3

ip2.3 op2.3 op3.3 op4.1

ip2.1 op2.1 op3.1

ip2.2 op2.2 op3.2

ip2.3 op2.3 op3.3

ip2.4 op2.4 op3.4

Details of Option Button FSM


StateChart for the Currency ConverterIn Storage

Idle

U .S . Dollar Amount SelectedCountry Selected Missing Country and Dollar Message

Missing U .S . Dollar Message Missing Country Message

Both Inputs Done

Equiv . Amount Displayed

Ip 2/op 2 ,op 3

Ip 1/op 1

Ip 3/op 5

Ip 3 /op 7Ip 3/ op 7

Ip 6Ip 6

Ip 1 /op 1

Ip 2/ op 2 ,op 3

Ip 1 /op 1Ip 2 /op 2 ,op 3

Ip 6

Ip 3 /op 8

Ip 4

GUI action

Executing

Ip 4___________________

op 2 , op 4, op 9 , op 10

ip 5 Quit End Application


Observations and Conclusions

• Classes/objects are MUCH more complicated than procedures - need to consider inheritance, polymorphism - (the encapsulation part is easy) - o-o design proceeds by composition, not decomposition • Complexity is moved from methods to messaging among objects - hence unit level testing becomes integration level testing - it's like that bump in the rug, you can push down on it, but it pops up somewhere else. • Models to describe integration level testing are inadequate (you can't test what you don't model)


Extended Definitions

• An MM-Path starts with a method and ends when it reaches a method which does not issue any messages of its own ( message quiescence) • Since MM-Paths are composed of linked method-message pairs in an object network, they interleave and branch off from other MM-Paths.

• An ASF begins with a port input event • This system level input triggers the method-message sequence of an MM-Path which may trigger other MM-Paths • The sequence of MM-Paths ends with a port output event (event quiescence)

An MM-Path in object-oriented software is a sequence ofmethod executions linked by messages.

An atomic system function (ASF ) is a sequence ofstatements that begins with an input port eventand ends with an output port event.


Port Input Event

Port Output Event

Data Place

Method Execution Path

Message Send/Return

Constructs for Event- and Message-Driven Petri Nets


object A

object B

Message from object A to object B


d1

d2

d3d5

d4

d6msg1

msg2

mep1

mep2

mep3

mep4

mep5

mep6

(Second Edition Cover Story)


s1

d1

p1

p10

s2

d2

p2

p15

p19

p23

s7

p7

p24

p25

p14

d1 d2

s6

p6

p26

d1 d2

p11

s3

d2

p3

p16

p20

p12

s4

d2

p4

p17

p21

p13

s5

d2

p5

p18

p22

s8

p8

p27

ASFs for the Currency Converter


s1

d1

p1

p23

s7

p7

p24

p25

p14

s6

p6

p26

p12

s4

d2

p4

p17

p21

EDPN Composition of four ASFs


s1

s3

s7

s6

s4 s5s2

s8

Directed Graph of Intended ASF Sequences


“Forced Navigation” Adjacency Matrix

00000000s810011111s711011111s600101110s500110110s400111010s300111100s200011111s1s8s7s6s5s4s3s2s1

software testing a craftmans approach

Documents