Meta-Analysis Matthew Burns University of Missouri Overview Resources https:// www.meta-analysis.com/downloads/Meta%20Analysis%20Fixed%20vs%20Random%20effec

ts.pdf https://www.meta-analysis.com/pages/formulas.php School Psychologists and Research

Keith (2008) three research activities Conducting Research Consuming Research Synthesizing Research

Most Famous Articles in our Field? Types of Errors in Narratively Synthesizing Research Selectivity Erroneous detailing Nonrecognition of faulty author conclusions

Suppression of contrary findings The larger the literature, the less useful reviews Center et al. (1995) study of Reading Recovery Reading Recovery Pretest

Posttest Clay Book Level Test Word Reading Test Neale Analysis of Reading Passage Reading Test Diagnostic Spelling Test Phonemic Awareness Test

Cloze Test Word Attack Skills Test .59 2.55 4.64 2.32

7.26 18.27 .68 Control

Pretest Posttest .47 .83 9.96

30.86 12.59 4.82 55.55 22.86 51.59 24.23

63.50 1.83 7.43 .17

2.57 2.77 12.17 1.40 5.73 22.67

39.27 23.17 41.33 17.47 34.47 SIGNIFICANCE = SIZE OF SAMPLE X

SIZE OF EFFECT A standardized estimate of the difference between experimental and control groups X1 X2

d= (s1 + s2)/2 2r d=

1-r2 Hedges g Hedges g Upwardly biased for small sample sizes Multiply g by (1-3/[N-9])

N total sample size Effect Size Websites http://www.campbellcollaboration.org/resour ces/effect_size_input.php http://www.cognitiveflexibility.org/effectsize/ http://www.lyonsmorris.com/ma1/

Interpreting ES Cohen (1988) .20 small .50 medium .80 large

Somewhat arbitrarily selected More (was) to be gained than lost by supplying a common conventional frame of reference (p. 25). PostTest d = -0.18

Pre to Post Test d = 0.76 Synthesizing Meta-Analysis

A methodology for systematically examining a body of research (Glass, 1976). Examine the effect of variables on the phenomenon of interest Exhaustive search Established and explicit inclusion criteria Report an empirically derived effect size

Integrate research to create stronger generalizations Meta-Analysis of Reading Recovery Reading Recovery All students

k d 16 .63

Discontinued 5 1.11

Not Discontinued 4 -.04 Interventions for first graders

All 26 .63 Other than RR *

(2000) 10 .64

Elbaum, Vaughn, Hughes, & Moody Interventions for Children with LD Reading comprehension Direct instruction Psycholinguistic training

Modality instruction Diet Perceptual training Kavale & Forness, 2000 1.13 .84

.39 .15 .12 .08 Contributions to Learning Hattie 2009 The student d = .40

The school d = .23 The teacher d = .49 The curriculum d = .45

Meta-Analysis Hattie 2009: Curriculum Vocabulary Instruction d = .67 Phonics Instruction d = .60 Comprehension Instruction d = .58 Whole Language d = .06 Perceptual-Motor d = .08

Teacher Roles: Hattie 2009 Activator Facilitator Feedback d = .72

Meta-cognition d = .67 Direct Instruction d = .59 Mastery Learning d = .57 Form Assess. d = .46 Simulation/game d = .32

Inquiry-based d = .31 Class size d = .21 Problem-based d = .15 Inductive teach d = .06

Total Total d = .60 d = .17

Why Conduct Meta-Analytic Research Conflicting results Clarity in a large literature Quantify relationships between variables across studies to build theory

Conflicting Results: Problem Solving Teams Mean Effect Sizes Clarity: Disability Simulations Theory

Need for Meta-Analyses PROCESS Beginning Identify your research questions broadly stated Identify key words that used to describe

relevant concepts in literature Develop inclusion criteria (For what are you looking?) Search Terms Comprehensive

Limit false positives, BUT almost eliminate false negatives Include synonyms Include vocabulary Based on literature!!!! IOA

Searching the Literature

Identify source for electronic search Conduct search Eliminate based on abstract Eliminate based on article Search reference lists Contact authors of found articles

Learner control AND reading AND math* AND literacy AND science AND social studies AND history AND language AND English language arts AND writing AND English language learner AND computer assisted learning AND classroom AND computers AND educational technology AND school

Codes Four general codes Substantive features of what is being studied Research methods Characteristics of the researcher and research context How the study is reported

Terms must be based on literature Clearly described in paper Coding manual pilot and IOA Type of Meta-Analysis Fixed Effects Random Effects

https:// www.meta-analysis.com/downloads/Meta-analysis%20Fixed-effect%2 0vs%20Random-effects%20models.pdf Fixed Effects Meta-Analysis Very similar studies same population etc. One true effect sizes

Use when all studies are identical and identified Five drug company studies Six studies on Reading Recovery Random Effects Model Articles are similar but vary somehow There is a distribution of effect sizes

Combined effect Fixed effects = the one common effect Random effects = mean of the population of true effects Effects weighting (more on that later) Random effects not as influenced by extremely small or large studies (part of population)

Variance for Random Effects v= n1 + n2 + g n1*n2

2*(n1 + n2) 2 Weighting Random Effects Q measure of total variance and homogeneity SCD Effect Size

Cohens d is not appropriate for SCD No-assumptions effect size (Busk & Serlin, 1992) Autocorrelation Percentage non-overlapping data (PND, Scruggs, Mastropieri, & Casto, 1987) No confidence intervals

Affected by outlying data Auto correlation r = .54 Auto correlation r = .57 Auto correlation r = .48

Total Auto correlation r = .57 Auto correlation r = .72 Non-Overlap of All Pairs (NAP) Parker & Vannest (2009) Based on area under the curve (AUC)

Compares overlap between each data point in A with each data point in B NAP = number of comparison pairs with no overlap divided by the total number of pairs Can be converted to other effect sizes How Large?

ROC with large effect based on visual analysis of multiple-baseline data = .96 Small effect = .90 Petersen-Brown, Karich, & Symons, 2012 Phi

Remember PND, PAND, NAP unknown distribution cannot convert to CI Phi commonly used effect size Cohen 1989 Converting to Phi NAP uses formula by Ruscio (2008)

AUC to phi Interpret Phi as ES

.10 = small .30 = medium .50 = large Derived from large-group designs Convert d to correlation with .20, .50, and .80

Threats to Validity Nonindependent effects Exclude repeats Average

Gaps in the literature Apples and oranges File drawer problem Fail-Safe N (Orwin, 1983) Nfs =

N0 (d0 dc) dc N0 = Number of studies d0 = Mean of observed effects

dc = Criterion d (e.g., .20)