Using the Rasch model to validate a multiple choice English achievement test

Rasch analyses have been linked by numerous scholars to the six facets of Messick’s (1989) concept of construct validity and are commonly used to evaluate pedagogical assessment. Compared to deterministic statistics from classical test theory, the Rasch model’s prescriptive methods have been argued to provide stronger validity evidence. Rasch-based methods estimate probabilities of item endorsements according to person ability and item difficulty parameters, highlighting items that produce some degree of unexpected response patterning. In the current article, a multiple-choice achievement test taken by English as a Foreign Language students at a private university in Japan was analyzed. The results show very little misfit to the Rasch model and that the level of the test was appropriately targeted to the abilities of the test-taking population, covering a range of statistically distinct difficulties. Fit statistics, an item-person map, item strata, Rasch measures, point-measure correlations and their relation to Messickian validity are discussed.