Developing a rating scale to assess phraseological competence

Tavakoli, P. ORCID: https://orcid.org/0000-0003-0807-3709, Uchihara, T., Smyth, P. ORCID: https://orcid.org/0000-0001-8799-6186, Mazhurnaya, S. and Koizumi, R. (2026) Developing a rating scale to assess phraseological competence. Research Methods in Applied Linguistics, 5 (3). 100366. ISSN 2772-7661 doi: 10.1016/j.rmal.2026.100366

Abstract/Summary

The study set out to design, develop, and validate a rating scale for assessing second language test-takers' phraseological competence. It also aimed to examine the relationship between human ratings of phraseological competence and assessed levels of proficiency in the Test of English for Educational Purposes (TEEP). The dataset comprises spoken performances of 127 TEEP candidates placed across six levels of proficiency. Four dimensions of accuracy/appropriacy, diversity, sophistication, and abundance were identified, and a scale of 1-6 was developed. Training was provided to three expert raters and two standardisation meetings were held before they were asked to rate the spoken performances independently. Many-facet Rasch Measurement was used to analyse three facets (i.e., candidates, raters, and criteria) and compute logit scores as indicators of test-takers' phraseological competence. The analysis showed that a 5-level scale worked more effectively, demonstrating adequate overall data fit to the Rasch model. The results suggested that both raters and criteria in the modified 5-level scale worked well and sufficiently contributed to the relatively precise and adequate measurement of phraseological competence for research and proficiency assessment purposes. The ANOVA results revealed a significant main effect of proficiency level on the ratings, implying that phraseological competence, represented by the logit scores, differed significantly across assessed level of proficiency. Areas for improvement in the measurement and implications for language testing are discussed.

Item Type	Article
URI	https://centaur.reading.ac.uk/id/eprint/131546
Identification Number/DOI	10.1016/j.rmal.2026.100366
Refereed	Yes
Divisions	Interdisciplinary Research Centres (IDRCs) > Centre for Literacy and Multilingualism (CeLM) Arts, Humanities and Social Science > School of Literature and Languages > English Language and Applied Linguistics Arts, Humanities and Social Science > Global Academy
Publisher	Elsevier
Download/View statistics	View download statistics for this item