An experimental assessment of module documentation-based testing

Salmi Baharom, Zarina Shukur

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

Context: Testing a module that has memory using the black-box approach has been found to be expensive and relatively ineffective. Instead, testing without knowledge of the specifications (white-box approach) may not be effective in showing whether a program has been properly implemented as stated in its specifications. We propose instead a grey-box approach called Module Documentation-based Testing or MD-Test, the heart of which is an automatic generation of the test oracle from the external and internal views of the module. Objective: This paper presents an empirical analysis and comparison of MD-Test against three existing testing tools. Method: The experiment was conducted using a mutation-testing approach, in two phases that assess the capability of MD-Test in general and its capability of evaluating test results in particular. Results: The results of the general assessment indicate that MD-Test is more effective than the other three tools under comparison, where it is able to detect all faults. The second phase of the experiment, which is significant to this study, compares the capabilities of MD-Test and JUnit-black using the test evaluation results. Likewise, an analysis of the test evaluation results shows that MD-Test is more effective and efficient, where MD-Test is able to detect at least the same number of faults as, or is at par with, the black-box approach. Conclusion: It is concluded that test evaluation using grey-box approach is more effective and efficient that the black-box approach when testing a module that has memory.

Original languageEnglish
Pages (from-to)747-760
Number of pages14
JournalInformation and Software Technology
Volume53
Issue number7
DOIs
Publication statusPublished - Jul 2011

Fingerprint

Testing
Specifications
Data storage equipment
Experiments

Keywords

  • Automated module testing
  • Grey-box testing
  • Mutation-testing
  • Specification-based testing

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Computer Science Applications

Cite this

An experimental assessment of module documentation-based testing. / Baharom, Salmi; Shukur, Zarina.

In: Information and Software Technology, Vol. 53, No. 7, 07.2011, p. 747-760.

Research output: Contribution to journalArticle

@article{ccc8815ae33b4bcd8e9c815bbf8f288e,
title = "An experimental assessment of module documentation-based testing",
abstract = "Context: Testing a module that has memory using the black-box approach has been found to be expensive and relatively ineffective. Instead, testing without knowledge of the specifications (white-box approach) may not be effective in showing whether a program has been properly implemented as stated in its specifications. We propose instead a grey-box approach called Module Documentation-based Testing or MD-Test, the heart of which is an automatic generation of the test oracle from the external and internal views of the module. Objective: This paper presents an empirical analysis and comparison of MD-Test against three existing testing tools. Method: The experiment was conducted using a mutation-testing approach, in two phases that assess the capability of MD-Test in general and its capability of evaluating test results in particular. Results: The results of the general assessment indicate that MD-Test is more effective than the other three tools under comparison, where it is able to detect all faults. The second phase of the experiment, which is significant to this study, compares the capabilities of MD-Test and JUnit-black using the test evaluation results. Likewise, an analysis of the test evaluation results shows that MD-Test is more effective and efficient, where MD-Test is able to detect at least the same number of faults as, or is at par with, the black-box approach. Conclusion: It is concluded that test evaluation using grey-box approach is more effective and efficient that the black-box approach when testing a module that has memory.",
keywords = "Automated module testing, Grey-box testing, Mutation-testing, Specification-based testing",
author = "Salmi Baharom and Zarina Shukur",
year = "2011",
month = "7",
doi = "10.1016/j.infsof.2011.01.005",
language = "English",
volume = "53",
pages = "747--760",
journal = "Information and Software Technology",
issn = "0950-5849",
publisher = "Elsevier",
number = "7",

}

TY - JOUR

T1 - An experimental assessment of module documentation-based testing

AU - Baharom, Salmi

AU - Shukur, Zarina

PY - 2011/7

Y1 - 2011/7

N2 - Context: Testing a module that has memory using the black-box approach has been found to be expensive and relatively ineffective. Instead, testing without knowledge of the specifications (white-box approach) may not be effective in showing whether a program has been properly implemented as stated in its specifications. We propose instead a grey-box approach called Module Documentation-based Testing or MD-Test, the heart of which is an automatic generation of the test oracle from the external and internal views of the module. Objective: This paper presents an empirical analysis and comparison of MD-Test against three existing testing tools. Method: The experiment was conducted using a mutation-testing approach, in two phases that assess the capability of MD-Test in general and its capability of evaluating test results in particular. Results: The results of the general assessment indicate that MD-Test is more effective than the other three tools under comparison, where it is able to detect all faults. The second phase of the experiment, which is significant to this study, compares the capabilities of MD-Test and JUnit-black using the test evaluation results. Likewise, an analysis of the test evaluation results shows that MD-Test is more effective and efficient, where MD-Test is able to detect at least the same number of faults as, or is at par with, the black-box approach. Conclusion: It is concluded that test evaluation using grey-box approach is more effective and efficient that the black-box approach when testing a module that has memory.

AB - Context: Testing a module that has memory using the black-box approach has been found to be expensive and relatively ineffective. Instead, testing without knowledge of the specifications (white-box approach) may not be effective in showing whether a program has been properly implemented as stated in its specifications. We propose instead a grey-box approach called Module Documentation-based Testing or MD-Test, the heart of which is an automatic generation of the test oracle from the external and internal views of the module. Objective: This paper presents an empirical analysis and comparison of MD-Test against three existing testing tools. Method: The experiment was conducted using a mutation-testing approach, in two phases that assess the capability of MD-Test in general and its capability of evaluating test results in particular. Results: The results of the general assessment indicate that MD-Test is more effective than the other three tools under comparison, where it is able to detect all faults. The second phase of the experiment, which is significant to this study, compares the capabilities of MD-Test and JUnit-black using the test evaluation results. Likewise, an analysis of the test evaluation results shows that MD-Test is more effective and efficient, where MD-Test is able to detect at least the same number of faults as, or is at par with, the black-box approach. Conclusion: It is concluded that test evaluation using grey-box approach is more effective and efficient that the black-box approach when testing a module that has memory.

KW - Automated module testing

KW - Grey-box testing

KW - Mutation-testing

KW - Specification-based testing

UR - http://www.scopus.com/inward/record.url?scp=79955065276&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79955065276&partnerID=8YFLogxK

U2 - 10.1016/j.infsof.2011.01.005

DO - 10.1016/j.infsof.2011.01.005

M3 - Article

AN - SCOPUS:79955065276

VL - 53

SP - 747

EP - 760

JO - Information and Software Technology

JF - Information and Software Technology

SN - 0950-5849

IS - 7

ER -