Results From G-Theory Analysis Note that the first six booklets were administered to students in the HSA sample, whereas the remaining seven booklets were administered to students in the MSA sample; the first and sixth booklets were administered in both designs as anchor booklets. Table 5 shows the aggregated variance components for these two student samples.
2.3 Aggregated Variance Components Estimates for HSA and MSA Samples Notably, the absolute values of the variance components are small because the ratings are dichotomous. However, the percentage of total variance in the ratings accounted for each component shows that the writing tasks functioned differently across the two samples.
On the one hand, it can be seen that the majority of variation of ratings in either sample is due to differences between students (24.3% and 8.4%, respectively), differences between tasks (9.0% and 37.3%, respectively), and the interaction between students and tasks (17.2% and 11.4%, respectively). However, the relative amount of variance explained by the main effect differs between the two samples. In the HSA sample, the amount of variation in the ratings that is due to students is almost three times as large as the amount of variation due to tasks. In the MSA sample, this effect is reversed and the amount of variation due to tasks is four times larger than the amount of variation due to students in this sample. As we see next, this difference is also captured in the results of the multifaceted Rasch modeling. Similar to Chiu and Wolfe (2002), the aggregated effects combine substantial variation across booklets highlighting the importance of comparing all available data instead of using just one particular booklet, for example.
In addition to the effects just discussed, we can also note that the amounts of unexplained variation are similar in both samples (25.2% and 19.0%, respectively) and that the main effects of different raters are negligible (1.2% and 2.2%, respectively). Among the interaction effects, the effect of the interaction between students, tasks, and raters is the second-largest one in both samples (8.6% and 7.1%, respectively). This shows that, on average, there are differences in mean ratings that are attributable to the fact that different raters are assigned to different student responses from different booklets, and tasks within a booklet are of different difficulty.
