On testing effectiveness of metamorphic relations

Software testing is a very crucial approach for assuring the quality of the software applications. In this century, software is in use in every sphere of life. Hence, it is very essential to guarantee the correctness of the software functionality to a certain degree. Absence of proper methods to assure the correctness of the software can lead to disasters. Many testing methods have been proposed to select test inputs such that faults in the program can be detected effectively. However, several programs are non-testable due to the so-called 'oracle problem'. The oracle problem means that either there does not exist a mechanism (called 'oracle') to verify the test output given any possible program input, or it is very expensive, if not impossible, to apply the oracle. Detecting faults in a program without the oracle is one of the major challenges of software testing. Metamorphic testing is an innovative approach to alleviating the oracle problem. In metamorphic testing, metamorphic relations are derived from the necessary characteristics of the system under test. These relations can help to generate test inputs and verify the correctness of the test outputs without the need of an oracle. Their effectiveness has a significant role in metamorphic testing. Hence, it is important to select effective metamorphic relations in metamorphic testing. Different metamorphic relations can generate test inputs with different code coverage. Thus, code coverage of generated test inputs can be considered as a characteristic of the corresponding metamorphic relation. This study aims to find out whether the fault-detection effectiveness of a metamorphic relation can be determined by its code coverage. Some case studies have been conducted to analyze the relationship between the code coverage and the fault-detection effectiveness of metamorphic relations. The systems for these studies were selected from different domains that require diverse ranges of metamorphic relations being derived. Moreover, efforts have been given to conduct the research in a platform independent way by choosing the systems from different platforms. Automated mutant generator tools have been used to generate mutants (faulty versions of the system under test) for ensuring non-biased mutants in the experiments. Fault-detection effectiveness of metamorphic relations is calculated by applying metamorphic testing on these mutants. Two basic coverage criteria namely line and branch coverage is taken into considerations. Automated code coverage tools are used to collect coverage data achieved by different metamorphic relations. A certain degree of correlation between the code coverage and fault-detection effectiveness of a metamorphic relation was observed. Code coverage attained by metamorphic relations was found to be a good indicator of the fault-detection effectiveness of these relations. Thus, the selection of metamorphic relations with higher code coverage can help metamorphic testing to become more efficient.