Complementary evidence in the early-stage validation of language tests: Classical Test Theory and Rasch analyses

Abstract

Test validation may more aptly be conceived of as the process of designing language tests responsibly. While a good test gains in reputation as it is administered over time, the early stages of its validation are perhaps the most critical. There is now general agreement that the validation process should be reported in the form of an argument that brings together multiple sets of evidence to justify the design and implementation of the measurement instrument, the language test. The format of such integration is, however, still contestable ground. Referring to an example of language test design and development, this paper seeks to demonstrate how a framework for responsible test design may be employed to achieve such an integrated argument, as well as how two of the methodological tools most frequently employed to muster empirical evidence for validating test design, namely classical test theory (CTT) and Rasch analyses, complement each other in designing tests responsibly. While most language tests designed in South Africa have used CTT, the employment of Rasch analyses has been more limited. A secondary aim of the paper is therefore to provide applied linguists who work in the subfield of language testing with an example of how the latter kind of analysis can complement the former. In all, however, these disparate approaches must be integrated into the theoretical justification for the development of language tests, in order to satisfy a number of conditions for their responsible design.