All posts by pamrevie

Research Methods, Research Methods / Methodological Features and Effect Sizes

RESEARCH Methods / Methodological Features and Effect Sizes

As evidence-based reform becomes increasingly important in educational policy, it is becoming essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. The purpose of this article is to examine how methodological features such as types of publication, sample sizes, and research designs affect effect sizes in experiments. A total of 645 studies from 12 recent reviews of evaluations of reading, mathematics, and science programs were studied. The findings suggest that effect sizes are roughly twice as large for published articles, small-scale trials, and experimenter-made measures, than for unpublished documents, large-scale studies, and independent measures, respectively.  In addition, effect sizes are significantly higher in quasi-experiments than in randomized experiments. Explanations for the effects of methodological features on effect sizes are discussed, as are implications for evidence-based policy.

Technical Report

Cheung, A., & Slavin, R. (2016). How methodological features affect effect sizes in education. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Cheung, A., & Slavin, R. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45 (5), 283-292.

Rigorous evidence of program effectiveness has become increasingly important with the 2015 passage of the Every Student Succeeds Act (ESSA). One question that has not yet been fully explored is whether program evaluations carried out or commissioned by developers produce larger effect sizes than evaluations conducted by independent third parties. Using study data from the What Works Clearinghouse, we find evidence of a “developer effect,” where program evaluations carried out or commissioned by developers produced average effect sizes that were substantially larger than those identified in evaluations conducted by independent parties. We explore potential reasons for the existence of a “developer effect” and provide evidence that interventions evaluated by developers were not simply more effective than those evaluated by independent parties. We conclude by discussing plausible explanations for this phenomenon as well as providing suggestions for researchers to mitigate potential bias in evaluations moving forward.

Technical Report

Wolf, R., Morrison, J.M., Inns, A., Slavin, R. E., & Risman, K. (2020). Average effect sizes in developer-commissioned and independent evaluations. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Wolf, R., Morrison, J.M., Inns, A., Slavin, R. E., & Risman, K. (2020). Average effect sizes in developer-commissioned and independent evaluations. Journal of Research on Educational Effectiveness. DOI: 10.1080/19345747.2020.1726537

Large-scale randomized studies provide the best means of evaluating practical, replicable approaches to improving educational outcomes. This article discusses the advantages, problems, and pitfalls of these evaluations, focusing on alternative methods of randomization, recruitment, ensuring high-quality implementation, dealing with attrition, and data analysis. It also discusses means of increasing the chances that large randomized experiments will find positive effects, and interpreting effect sizes.

Technical Report

Slavin, R., & Cheung, A. (2017). Lessons learned from large-scale randomized experiments. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Slavin, R., & Cheung, A. (2017). Lessons learned from large-scale randomized experiments.Journal of Education for Students Placed at Risk. Doi: http://dx.doi.org/10.1080/10824669.2017.1360774

Program effectiveness reviews in education seek to provide educators with scientifically valid and useful summaries of evidence on achievement effects of various interventions. Different reviewers have different policies on measures of content taught in the experimental group but not the control group, called here treatment-inherent measures. These are contrasted with treatment-independent measures of content emphasized equally in experimental and control groups. The What Works Clearinghouse (WWC) averages effect sizes from such measures with those from treatment-independent measures, while the Best Evidence Encyclopedia (BEE) excludes treatment-inherent measures. This article contrasts effect sizes from treatment-inherent and treatment-independent measures in WWC reading and math reviews to explore the degree to which these measures produce different estimates. In all comparisons, treatment-inherent measures produce much larger positive effect sizes than treatment-independent measures. Based on these findings, it is suggested that program effectiveness reviews exclude treatment-inherent measures, or at least report them separately.

Technical Report

Slavin, R.E., & Madden, N.A. (2011). Measures inherent to treatments in program effectiveness reviews. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.slavin-madden-2009-measures-inherent-to-treatments

Published Report

Slavin, R.E., & Madden, N.A. (2011). Measures inherent to treatments in program effectiveness reviews. Journal of Research on Educational Effectiveness, 4 (4), 370-380.

Research in fields other than education has found that studies with small sample sizes tend to have larger effect sizes than those with large samples.  This article examines the relationship between sample size and effect size in education. It analyzes data from 185 studies of elementary and secondary mathematics programs that met the standards of the Best-Evidence Encyclopedia. As predicted, there was a significant negative correlation between sample size and effect size. The differences in effect sizes between small and large experiments were much greater than those between randomized and matched experiments. Explanations for the effects of sample size on effect size are discussed.

Technical Report

Slavin, R.E., & Smith, D. (2009).  The relationship between sample sizes and effect sizes in systematic reviews in education.  Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Slavin, R.E., & Smith, D. (2009).  The relationship between sample sizes and effect sizes in systematic reviews in education.  Educational Evaluation and Policy Analysis, 31 (4), 500-506.

Early Childhood / Early Childhood Education

Early Childhood / Early Childhood Education

This systematic review of research on early childhood programs seeks to identify effective approaches capable of improving literacy and language outcomes for preschoolers. It applies consistent standards to determine the strength of evidence supporting a variety of approaches, which fell into two main categories: balanced approaches, which include phonemic awareness, phonics, and other skills along with child-initiated activities, and developmental-constructivist approaches that focus on child-initiated activities with little direct teaching of early literacy skills. Inclusion criteria included use of randomized or matched control groups, evidence of initial equality, a minimum study duration of 12 weeks, and valid measures of literacy and language. Thirty-two studies evaluating 22 programs found that early childhood programs that have a balance of skill-focused and child-initiated activities programs had significant evidence of positive literacy and language outcomes at the end of preschool and on kindergarten follow-up measures. Effects were smaller and not statistically significant for developmental-constructivist programs.

Technical Report

Chambers, B., Cheung, A., & Slavin, R. (2016). Literacy and language outcomes of balanced and developmental-constructivist approaches to early childhood education: A systematic review.Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University

Published Report

Chambers, B., Cheung, A., & Slavin, R. (2016). Literacy and language outcomes of balanced and developmental-constructivist approaches to early childhood education: A systematic review.Educational Research Review 18, 88-111.

Comprehensive School Reform/K-12 Meta-Analysis (Borman)

K-12 Meta-Analysis (Borman)

Which comprehensive school reform programs have been proven to help elementary and secondary students achieve? To find out, this review summarizes evidence on comprehensive school reform (CSR) models in elementary and secondary schools. Comprehensive school reform models are programs used schoolwide to improve student achievement. They typically include the following elements:

  • Innovative approaches to instruction and curriculum used in many subjects throughout the school
  • Extensive, ongoing professional development, and coaches or facilitators in the building to help manage the reform process
  • Measurable goals and benchmarks for student achievement
  • Emphasis on parent and community involvement

CSR models are developed and supported by national organizations, mostly nonprofits, that provide professional development, materials, and support to networks of schools.

Published Report

Borman, G., Hewes G., Overman, L., & Brown, S. (2003, Summer). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73, 2, 125-230.
Science / Secondary

Science / Secondary

High school students using microscope, examining slides in science class

This article reports a systematic review of research on science programs in grades 6-12. Twenty-one studies met inclusion criteria including use of randomized or matched assignment to conditions, measures that assess content emphasized equally in experimental and control groups, and a duration of at least 12 weeks. Programs fell into four categories. Instructional process programs (ES=+0.24) and technology programs (ES=+0.47) had positive sample-size weighted mean effect sizes, while use of science kits (ES=+0.05) and innovative textbooks (ES=+0.10) had much lower effects. Outcomes support the use of programs with a strong focus on professional development, technology, and support for teaching, rather than materials-focused innovations.

Technical Report

Cheung, A., Slavin, R.E., Kim, E., & Lake, C. (2015). Effective secondary science programs: A best-evidence synthesis. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Cheung, A., Slavin, R.E., Kim, E., & Lake, C. (2016). Effective secondary science programs: A best-evidence synthesis. Journal of Research on Science Teaching, 54 (1), 58-81. Doi: 10.1002/tea.21338

SCIENCE / ELEMENTARY

SCIENCE / ELEMENTARY

Which science programs have been proven to help elementary students to succeed? To find out, this review summarizes evidence on three types of programs designed to improve the science achievement of students in grades K–6:

  • Inquiry-oriented programs without science kits, such as Increasing Conceptual Challenge, Science IDEAS, and Collaborative Concept Mapping. These programs help teachers learn and use generic processes, such as cooperative learning, concept development, and science-reading integration, in their daily science teaching.
  • Inquiry-oriented programs with science kits, such as Insights, FOSS, STC, SCALE, and Teaching SMART. The theory of action in science kit programs is that implementing hands-on activities helps to build deep learning about the scientific process and core concepts of elementary science.
  • Technology programs, such as BrainPOP, The Voyage of the Mimi, and web-based labs. Technologies utilized in these approaches include computer-assisted instruction and class-focused technology (such as video and interactive whiteboard technologies).

The evidence from studies that met the review’s inclusion criteria supports a view that improving outcomes in elementary science depends on improving teachers’ skills in presenting lessons, engaging and motivating students, and integrating science and reading. Technology applications that help teachers teach more compelling lessons and that use video to reinforce lessons also have promise.

Technical Report

Slavin, R.E., Lake, C., Hanley, P., & Thurston, A. (2012, May). Effective Programs for Elementary Science: A Best-Evidence Synthesis. Baltimore, MD: Johns Hopkins University, Center for Research and Reform in Education.

Published Report

Slavin, R. E., Lake, C., Hanley, P., & Thurston, A. (2014). Experimental evaluations of elementary science programs: A best-evidence synthesis. Journal of Research in  Science Teaching, 51 (7), 870-901.

Writing, Writing / Grades 2-12

Writing / Grades 2-12

This paper reviews research on outcomes of writing programs for students in grades 2 to 12. Studies had to meet rigorous standards of research including use of randomized or wellmatched control groups, measures independent of the program developers, researchers, and teachers, and adequate sample size and duration. Fourteen studies of 12 programs met the standards. Twelve (86%) were randomized, two matched. Programs were divided into three categories. Student achievement effects on writing were positive on average in all categories (Effect Size=+0.18), with similar outcomes for writing programs focused on the writing process (ES=+0.17), those using cooperative learning (ES=+0.16), and those focusing on interactions between reading and writing (ES=+0.19).

Technical Report

Slavin, R.E., Lake, C., Inns, A., Baye, A., Dachet, D., & Haslam, J. (2019). A quantitative synthesis of research on writing approaches in grades 2 to 12. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Slavin, R.E., Lake, C., Inns, A., Baye, A., Dachet, D., & Haslam, J. (2019). A quantitative synthesis of research on writing approaches in grades 2 to 12. London: Education Endowment Foundation.

READING / EFFECTIVENESS OF TECHNOLOGY

READING / EFFECTIVENESS OF TECHNOLOGY

The Center for Research and Reform in Education has completed reviews of educational technology and reading achievement, focusing on the effects of technology use on reading achievement in K-12 classrooms, and also on reading outcomes for struggling readers. The reviews can be accessed from the following links:

Effects of Technology Applications on Reading Achievement in K-12 Classrooms

Technical Report

Cheung, A., & Slavin, R.E. (2012). The Effectiveness of Educational Technology Applications for Enhancing Reading Achievement in K-12 Classrooms: A Meta-Analysis. Baltimore, MD: Johns Hopkins University, Center for Research and Reform in Education.

Published Report

Cheung, A., & Slavin, R.E. (2012). How features of educational technology programs affect student reading outcomes: A meta-analysis. Educational Research Review, 7 (3), 198-215. Doi:10.1016/j.edurev.2012.05.002

Effects of Technology Applications on Reading Outcomes for Struggling Readers

Technical Report

Cheung, A., & Slavin, R. E. (2013). Effects of educational technology applications on reading outcomes for struggling readers: A best-evidence synthesis.

Published Report

Cheung, A., & Slavin, R. E. (2013). Effects of educational technology applications on reading outcomes for struggling readers: A best-evidence synthesis. Reading Research Quarterly, 48 (3), 277-299.

READING / STRUGGLING READERS

Reading / Struggling Readers

This article reviews research on the outcomes of diverse reading programs on the achievement of struggling readers in elementary schools. Sixty-five studies of 51 different programs met rigorous standards. Eighty-three percent were randomized experiments and 17% quasi-experiments. Outcomes were positive for one-to-one tutoring and were positive but not as large for one-to-small group tutoring. There were no differences in outcomes between teachers and teaching assistants as tutors. Whole-class approaches (mostly cooperative learning) and whole-class/whole-school approaches incorporating tutoring for struggling readers obtained outcomes for struggling readers as large as those found for all forms of tutoring, on average, and benefitted many more students. Technology-supported adaptive instruction did not have significant positive outcomes for struggling readers, however. In agreement with previous reviews, this synthesis found that substantial impacts can be obtained for struggling readers with interventions aligned within a response to intervention network.

Technical Report

Neitzel, A., Lake, C., Pellegrini, M., & Slavin, R. (2021). A synthesis of quantitative research on programs for struggling readers in elementary schools. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Neitzel, A., Lake, C., Pellegrini, M., & Slavin, R. (in press). A synthesis of quantitative research on programs for struggling readers in elementary schools. Reading Research Quarterly 57 (1), 149-179. doi:10.1002/rrq.379

READING / ENGLISH LANGUAGE LEARNERS

READING / ENGLISH LANGUAGE LEARNERS

The Center for Research and Reform in Education has completed a review of reading programs for English language learners, which can be accessed from the link below.

Technical Report

Cheung, A., & Slavin, R. E. (2012). Effective reading programs for Spanish-dominant English language learners (ELLs) in the elementary grades: A synthesis of research. Baltimore, MD: Center for Research and Reform in Education, Johns Hopkins University.

Published Report

Cheung, A., & Slavin, R. E. (2012). Effective reading programs for Spanish-dominant English language learners (ELLs) in the elementary grades.Review of Educational Research, 82 (4), 351-395.