Assessing learner\'s writing skills according to CEFR scales
Multifaceted Rasch Modeling Generally speaking, the main focus of multifaceted modeling is to use a latent-variable measurement model to adjust the proficiency estimates of the students due to the design facets that have given rise to their variability. Recall that the writing tasks were designed to capture variation in student proficiencies across a single proficiency scale that was supposed to be used for reporting purposes. Because reading and listening comprehension tasks had been successfully scaled with a Rasch model from item response theory, and because the available sample size per task prohibited a reliable estimation of more complex models, we used a multifaceted Rasch model with a single proficiency variable. Our model included task, rater, and rating criterion as separate facets in addition to the student proficiency variable:
(2)where θsdenotes the latent student proficiency variable for writing and the subscripted β parameters denote the respective main or interaction effects for t = tasks, r = raters, and c = rating criteria. We fit a sequence of nested models to the data using the software ACER ConQuest to determine the best-fitting model, relatively speaking. The comparison of nested models was done using the deviance statistic, which is the difference in −2 log-likelihoods between nested models. It follows a chi-square distribution with degrees of freedom equal to the difference in estimated parameters between the two models. Furthermore, we evaluated the absolute fit of the final model that was determined through the sequential testing by inspecting the weighted sums-of-squares of the outfit statistics and their associated z-statistic produced for each effect in the model (see Adams & Wu, 2009, for a general framework).
RESULTS In this section we describe and synthesize key results from the descriptive analyses, the g-theory analyses, and the multifaceted Rasch analyses.
