Mittwoch Mai 27, 2009

Best Practices For Software Test Code

Best Practices For Software Test Code

Author: Joerg Sievers – Reviewers: Helge Delfs, Venkatesh Devarajan, Arun Kumar S, Srinivas Kg, Mario Schroeder, Elizabeth Matthis – Revision 10 – Last change: 1. July 2009

Test code is no production code

A well skilled Java developer isn't also a well skilled Java test code developer. The focus of a tester is different to a programmer and also automated test cases are different from methods of production code. They can be written in the same language and they maybe look alike but their goals are different. UI-tests should be more like an acceptance test because it is the same view a customer also has.

Explanation: In desktop environments the production code goal could be not to crash under the most worst circumstances. In test code you should stop working (=controlled, well reported stop of the test case execution) if an error occurs because the subsequent steps would report undefined results. It is easier to get an overview what happened if you only see one, the first error and not hundreds more which depended on the first one.

In web environments the production code goal could be to cloud the executed code in the browser. In test cases the feedback should be best traceable to reproduce simple steps if an error occurs.

Set preconditions

Be sure that a evadable default in the application is really the default in all environments. Otherwise you have to set the settings you need in front of the test case (preconditions). [OOO1]

An automated test case should not depend on other test cases

If you need preconditions to execute a test scenario you have to put in the precondition phase which always runs in front of any test case.

A test case needs to be re-runnable

No test pieces which being created while the test is running should hinder a re-run of the test case. If so, you have to remove such pieces in the precondition phase (=”running it always on the greenfield”). You should write automated software test not user scenario tests with undefined states – that would be another type of tests. Every testcase should have 3 sections: Entry, Action, CleanUp. Entry section will be useful to do any specific pre-settings, Action section is where actual testcase runs, CleanUp section is to undo all the actions performed by that particular testcase.

Use global tool settings and routines instead of own created

If an output path, a timeout, an environment variable, or also a helper method is defined globally it should be used instead of creating them in the test case on your own. This is the same for production and for test code to optimize the maintainability.

Explanation: Especially timeouts often defined in many test cases instead of using divisions or multiplications of the global timeout which is set in the most test tools.

Write test cases readable from top-level like use-cases

In production code the object-oriented approach is best practice. In test code, the focus is to simplify the work-flow of steps executed multiple times. The helper classes and abstraction layers should be made based on this focus but always it should be possible to read the important steps to get a test case done on the top-level and not in sub-routines.[MM01]

Useful feedback

A test case is a step-by-step instruction of a work-flow. In production code, feedback is not given if an action was successful. In test code you should do that more often to be able for others to reproduce these steps manually – and not only those which failed. [MF01]

Use defined prefixes for variables

Modern Integrated Development Environments (IDE) are able to show the type of a variable but for a better reading it makes sense to use defined prefixes. [OOO1] If there is no definition made the “ Hungarian Notation” [ Link English] [ Link German 1] can help.


prefix

type

i or n

integer

i

long

d

single

d

double

s

string

b

boolean

The following prefixes must be combined with an additional prefix:

a

array

g

global (from global.inc)

l

list (for variables declared for use of functions from t_list.inc)

Other naming schemes:

t

testcase

s

sub

f

function


Use clear indicators to show when an action is complete.

When an action has been started and you are waiting for the results, you have to have clear indicators for automated software to show when the action has been completed successfully. If you do not have such an indicator, then you have to use "timeouts", but they are always only a last resort.

For example: You want to load a document. To show that the document has been loaded, the status bar of the application changes or certain buttons become active.

Bibliography

OOO1: Thorsten Bosbach, Helge Delfs, Joerg Sievers, Joerg Skottke, OpenOffice.org Cookbook, 2004

MM01: Marc Michaelis, Boon and Bane of GUI Test Automation, 2008

MF01: Mark Fewster, Dorothy Graham, Software Test Automation, 1999





1The German Wikipedia version has some more tables and descriptions


Freitag Nov 14, 2008

Hi Admin, Gnome is gone! We haven't done anything!

Storyboard

I worked at home on my broadband connected Sun Ray and the phone rang. A colleague was on the other side and said that their test zone has no GNOME anymore. He promised that they haven't done anything. It is just gone. The test zone is being used for installing and testing StarOffice (SO), OpenOffice.org (OOo) and StarOffice PDF Conversion Server (Converter). After promises that no other software than SO, OOo or the Converter have been installed in the last days, I tried to analyze the problem....

Analyze phase

root# svcs -x
svc:/application/font/fc-cache:default (FontConfig Cache Builder)
 State: maintenance since Wed Nov 12 22:39:44 2008
Reason: Start method failed repeatedly, last died on Killed (9).
   See: http://sun.com/msg/SMF-8000-KS
   See: fc-cache(1M)
   See: /var/svc/log/application-font-fc-cache:default.log
Impact: \*This service is not running.\*

Me: Upps, have you manipulate the system?
He: Ehm... I remember that I have installed and removed a child workspace and also I removed with prodreg a printer device driver I have needed to evaluate a customer problem.
root# more /var/svc/log/application-font-fc-cache:default.log

(...)
[ Mar 25 09:46:27 Executing start method ("/usr/bin/fc-cache") ]
[ Mar 25 09:46:41 Method "start" exited with status 0 ]
[ Nov 12 14:53:47 Stopping because service disabled. ]
[ Nov 12 14:53:47 Method property group 'stop' is not present. ]
[ Nov 12 14:54:47 Enabled. ]
[ Nov 12 14:54:50 Executing start method ("/usr/bin/fc-cache") ]
ld.so.1: fc-cache: fatal: libexpat.so.0: open failed: No such file or directory
[ Nov 12 14:54:50 Method "start" failed due to signal KILL ]

Me: The system has been manipulated by one of the software under test (SUT) you have removed. It has taken the libexpat-library with it. I will come back to you. [click]
A comparison with sister-system and I was sure that it is a specific problem to this test-system.

Resolving the problem

I copied the library and created the symbolic links to get the following result:
root# ls -lia libexpat\*
   1062318 lrwxrwxrwx   1 root     root          17 Nov 13 10:40 libexpat.so -> libexpat.so.0.5.0
   1062317 lrwxrwxrwx   1 root     root          17 Nov 13 10:39 libexpat.so.0 -> libexpat.so.0.5.0
   1062316 -r-xr-xr-x   1 root     bin       307540 Nov 13 10:37 libexpat.so.0.5.0

Now some commands
root# svcadm clear svc:/application/font/fc-cache:default
root# svcadm enable svc:/application/font/fc-cache:default
root# svcs -x
root# more /var/svc/log/application-font-fc-cache:default.log

to see that the service is back again:
[ Nov 13 10:45:07 Leaving maintenance because clear requested. ]
[ Nov 13 10:45:07 Enabled. ]
[ Nov 13 10:45:07 Executing start method ("/usr/bin/fc-cache") ]

Ringing up the colleague...
Me: Try to log-in again and tell me that GNOME isn't gone....
He: Yes, Gnome is back. Thank you!
Me: Please validate which nice piece of software removed your library that not customer will have the same problem. Thank you.

Dienstag Jan 16, 2007

Enhancing automated software test environment on OOo

The automated testing team inside Sun Microsystems maintains at the moment 100% of the GUI testing code of OOo but there is hope that also others will start writing test code in future which can be also shared and used by others. That would be a great test case development.
To get others into the boat we want to get a good set of test scripts [1] and will do some enhancements in these areas:

  1. Documentation
  2. Reuse
  3. Structured
  4. Mainenance

We know for 1.) that we need to document how to add a new L10N (=localization) language to the framework and also we need to make it easier (4.).

If you have suggestions for the other areas we should think about, subscribe to dev@qa.openoffice.org mailing list and let us know what your barrier is we need to break down.

We want to collect in the next three weeks which enhancements we need to to do to get the barriers down and you into the boat. I will summarize it and maybe you are alos interested to do some work with us...



[1]
Software Test Automation, Mark Fewster & Dorothy Graham, Addison Wesley, 1999, ISBN 0-201-33140-3, page 69; table 3.1 "Comparison of good and poor sets of test scripts that perform the same set of test cases"

About

jogi

Search

Archives
« April 2014
MoDiMiDoFrSaSo
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Heute
Bookmarks