2 authors: Gina Biancarosa University of Oregon 49

Large-Scale Studies: A Cautionary Note Although e-reading technology offers real promise for improving literacy outcomes, evidence of its effectiveness is relatively limited. As of early 2012, out of 321 literacy- intervention programs reviewed by the What Works Clearinghouse over a decade, only thirteen relied on e-reading technology to some extent.⁵⁵ Of these, six were deemed to have at least potentially positive effects with no overriding contrary evidence, but both the number of studies of the six interventions and the overall sample sizes for each were
generally small. Only Read 180 in grades four through nine and SuccessMaker in grades four through ten had a medium to large research base; both had small positive effects on reading comprehension.⁵⁶

In fact, only two large-scale studies of

e-reading technology tools have been con- ducted as of early 2012; thus we review them in detail here. Both provide sobering evidence that should temper excitement about rapidly advancing technological innovations and thus emphasize the importance of explicitly and thoroughly evaluating effectiveness, as well as the importance of considering what promotes full implementation.

In 2009, the Institute for Educational Sciences released findings from a federally funded

randomized control study that investigated the effectiveness of ten reading and mathematical software programs used in first- and fourth- grade classrooms.⁵⁷ Researchers measured outcomes by comparing student scores on state-mandated standardized tests in class- rooms where the programs were integrated with the curriculum with scores in classrooms where the programs were not used. Only one reading program resulted in statistically significantly improved outcomes in fourth grade, and these effects were small and not evident until its second year of use. None of the other reading or math programs led to significant differences in scores when com- pared with the “business as usual” instruc- tional programs.⁵⁸

In another federally funded, large-scale, randomized control trial published in 2011, researchers investigated Thinking Reader—an e-reading computer program for nine children’s novels that provides instruction, guided practice, and feedback to readers at one of five teacher-chosen individualized levels of support. The study

compared outcomes of sixth-grade students who participated in the intervention with those of control students who received regular instruction and found no significant differences.⁵⁹

In short, the two studies provide no evidence that large-scale implementation of e-reading technology improves educational outcomes. But they do raise issues that should be addressed in ongoing research into the effectiveness of the technology. The first study, for example, evaluated programs that used very different approaches to instruction, making it unclear whether the failure to find effects for most programs was attributable to the technology or to the instructional approach. Nor was it clear whether the

programs under study were complementary to and connected with daily instruction in treatment classrooms—a particularly impor- tant consideration in making sustained, purposeful, and effective use of the technol- ogy to improve reading. Neither was it clear how faithfully the programs were imple- mented in the intervention classrooms.
Because schools and districts were selected precisely for their inexperience with such tools, lack of experience and discomfort with technology may also have contributed to the predominantly null findings.

The Thinking Reader study raised another important issue by gathering data on how students used the program. It found that the frequency of use was nowhere near suggested levels—about 60 minutes a week rather

than the recommended 110 to 165 minutes. And although Thinking Reader designers recommend that students participating in the program read multiple novels, the study found that by the end of the school year, 12 percent of students had not even begun a novel, 20 percent had not finished their first novel, 31 percent had completed only one, and only 7 percent had completed a third.

One explanation for the failure of large-scale studies to find evidence that e-reading tech- nology is effective may thus be that positive outcomes depend as much on genuinely engaging teachers and their students in the use of e-reading tools as on the availability of the technology itself. Whereas efficacy trials of programs and devices tend to target eager users by default, generating positive out- comes in large-scale studies and in the field may require more concerted attention to how these tools can be made appealing and useful to less-than-optimally eager and knowledge- able users.