Meta-Analysis Matthew Burns University of Missouri Overview Resources

Meta-Analysis Matthew Burns University of Missouri Overview Resources

Meta-Analysis Matthew Burns University of Missouri Overview Resources https:// www.meta-analysis.com/downloads/Meta%20Analysis%20Fixed%20vs%20Random%20effec

ts.pdf https://www.meta-analysis.com/pages/formulas.php School Psychologists and Research

Keith (2008) three research activities Conducting Research Consuming Research Synthesizing Research

Most Famous Articles in our Field? Types of Errors in Narratively Synthesizing Research Selectivity Erroneous detailing Nonrecognition of faulty author conclusions

Suppression of contrary findings The larger the literature, the less useful reviews Center et al. (1995) study of Reading Recovery Reading Recovery Pretest

Posttest Clay Book Level Test Word Reading Test Neale Analysis of Reading Passage Reading Test Diagnostic Spelling Test Phonemic Awareness Test

Cloze Test Word Attack Skills Test .59 2.55 4.64 2.32

7.26 18.27 .68 Control

Pretest Posttest .47 .83 9.96

30.86 12.59 4.82 55.55 22.86 51.59 24.23

63.50 1.83 7.43 .17

2.57 2.77 12.17 1.40 5.73 22.67

39.27 23.17 41.33 17.47 34.47 SIGNIFICANCE = SIZE OF SAMPLE X

SIZE OF EFFECT A standardized estimate of the difference between experimental and control groups X1 X2

d= (s1 + s2)/2 2r d=

1-r2 Hedges g Hedges g Upwardly biased for small sample sizes Multiply g by (1-3/[N-9])

N total sample size Effect Size Websites http://www.campbellcollaboration.org/resour ces/effect_size_input.php http://www.cognitiveflexibility.org/effectsize/ http://www.lyonsmorris.com/ma1/

Interpreting ES Cohen (1988) .20 small .50 medium .80 large

Somewhat arbitrarily selected More (was) to be gained than lost by supplying a common conventional frame of reference (p. 25). PostTest d = -0.18

Pre to Post Test d = 0.76 Synthesizing Meta-Analysis

A methodology for systematically examining a body of research (Glass, 1976). Examine the effect of variables on the phenomenon of interest Exhaustive search Established and explicit inclusion criteria Report an empirically derived effect size

Integrate research to create stronger generalizations Meta-Analysis of Reading Recovery Reading Recovery All students

k d 16 .63

Discontinued 5 1.11

Not Discontinued 4 -.04 Interventions for first graders

All 26 .63 Other than RR *

(2000) 10 .64

Elbaum, Vaughn, Hughes, & Moody Interventions for Children with LD Reading comprehension Direct instruction Psycholinguistic training

Modality instruction Diet Perceptual training Kavale & Forness, 2000 1.13 .84

.39 .15 .12 .08 Contributions to Learning Hattie 2009 The student d = .40

The school d = .23 The teacher d = .49 The curriculum d = .45

Meta-Analysis Hattie 2009: Curriculum Vocabulary Instruction d = .67 Phonics Instruction d = .60 Comprehension Instruction d = .58 Whole Language d = .06 Perceptual-Motor d = .08

Teacher Roles: Hattie 2009 Activator Facilitator Feedback d = .72

Meta-cognition d = .67 Direct Instruction d = .59 Mastery Learning d = .57 Form Assess. d = .46 Simulation/game d = .32

Inquiry-based d = .31 Class size d = .21 Problem-based d = .15 Inductive teach d = .06

Total Total d = .60 d = .17

Why Conduct Meta-Analytic Research Conflicting results Clarity in a large literature Quantify relationships between variables across studies to build theory

Conflicting Results: Problem Solving Teams Mean Effect Sizes Clarity: Disability Simulations Theory

Need for Meta-Analyses PROCESS Beginning Identify your research questions broadly stated Identify key words that used to describe

relevant concepts in literature Develop inclusion criteria (For what are you looking?) Search Terms Comprehensive

Limit false positives, BUT almost eliminate false negatives Include synonyms Include vocabulary Based on literature!!!! IOA

Searching the Literature

Identify source for electronic search Conduct search Eliminate based on abstract Eliminate based on article Search reference lists Contact authors of found articles

Learner control AND reading AND math* AND literacy AND science AND social studies AND history AND language AND English language arts AND writing AND English language learner AND computer assisted learning AND classroom AND computers AND educational technology AND school

Codes Four general codes Substantive features of what is being studied Research methods Characteristics of the researcher and research context How the study is reported

Terms must be based on literature Clearly described in paper Coding manual pilot and IOA Type of Meta-Analysis Fixed Effects Random Effects

https:// www.meta-analysis.com/downloads/Meta-analysis%20Fixed-effect%2 0vs%20Random-effects%20models.pdf Fixed Effects Meta-Analysis Very similar studies same population etc. One true effect sizes

Use when all studies are identical and identified Five drug company studies Six studies on Reading Recovery Random Effects Model Articles are similar but vary somehow There is a distribution of effect sizes

Combined effect Fixed effects = the one common effect Random effects = mean of the population of true effects Effects weighting (more on that later) Random effects not as influenced by extremely small or large studies (part of population)

Variance for Random Effects v= n1 + n2 + g n1*n2

2*(n1 + n2) 2 Weighting Random Effects Q measure of total variance and homogeneity SCD Effect Size

Cohens d is not appropriate for SCD No-assumptions effect size (Busk & Serlin, 1992) Autocorrelation Percentage non-overlapping data (PND, Scruggs, Mastropieri, & Casto, 1987) No confidence intervals

Affected by outlying data Auto correlation r = .54 Auto correlation r = .57 Auto correlation r = .48

Total Auto correlation r = .57 Auto correlation r = .72 Non-Overlap of All Pairs (NAP) Parker & Vannest (2009) Based on area under the curve (AUC)

Compares overlap between each data point in A with each data point in B NAP = number of comparison pairs with no overlap divided by the total number of pairs Can be converted to other effect sizes How Large?

ROC with large effect based on visual analysis of multiple-baseline data = .96 Small effect = .90 Petersen-Brown, Karich, & Symons, 2012 Phi

Remember PND, PAND, NAP unknown distribution cannot convert to CI Phi commonly used effect size Cohen 1989 Converting to Phi NAP uses formula by Ruscio (2008)

AUC to phi Interpret Phi as ES

.10 = small .30 = medium .50 = large Derived from large-group designs Convert d to correlation with .20, .50, and .80

Threats to Validity Nonindependent effects Exclude repeats Average

Gaps in the literature Apples and oranges File drawer problem Fail-Safe N (Orwin, 1983) Nfs =

N0 (d0 dc) dc N0 = Number of studies d0 = Mean of observed effects

dc = Criterion d (e.g., .20)

Recently Viewed Presentations

  • CS434/534: introduction - Yale University

    CS434/534: introduction - Yale University

    Assigning servers to service should be independent of network topology. Configure server with whatever IP address the service expects. VM keeps the same IP address even after migration. Uniform high capacity: Maximum rate of server to server traffic flow should...
  • Genetic Engineering

    Genetic Engineering

    13-1 Genetic Engineering Selective Breeding Humans use the variations in plants, animals, and other organisms to pass down the desirable traits Examples: Dogs, livestock (cows, chickens), plants we eat Hybridization Crossing dissimilar individuals to bring together the best of both...
  • Button Gwinnett - cpb-us-e1.wpmucdn.com

    Button Gwinnett - cpb-us-e1.wpmucdn.com

    Briefly commander of Georgia's Continental battalion (replaced by Lachlan McIntosh) Button Gwinnett's Plantation on St. Catherine's Island. Lyman Hall. GA representative, who signed the Declaration of Independence. 12th governor of GA.
  • Day 1 Handouts: 1) Blooms taxonomy 2) Learning

    Day 1 Handouts: 1) Blooms taxonomy 2) Learning

    Levels of learning goals Next Activity: Work on your learning goals with your table group Testing achievement of learning goals: Slide 17 The Montillation of Traxoline when assessment goes astray Slide 19 Slide 20 Slide 21 Ideas for implementation: Slide...
  • Academic Entrepreneurship - Startsida

    Academic Entrepreneurship - Startsida

    Academic Entrepreneurship
  • Active Learning: Materials

    Active Learning: Materials

    For example, empty bottles, old utensils, cardboard tubes, bubble wrap, fabric and yarn, eyeglass cases, old tools, cookie tins, shoes, gloves, and so forth. Collecting materials can also make for a great scouting project or volunteer activity.
  • Creation D'Un Service Commun Sante Et Securite Au Travail

    Creation D'Un Service Commun Sante Et Securite Au Travail

    Pas de convention à ce jour pour travailler pour un service mutualisé. Collaboration à définir sur des missions transversales (nécessité de plusieurs conventions) En théorie, iln'est pas possible de solliciter l'ACFI sur des projets transversaux pour le service commun.
  • Their - Capital City Beekeepers Association

    Their - Capital City Beekeepers Association

    Shore up weak hive, make another nuc, help a friend, etc. Let them make honey (watch for plugging) Eat it, Store it, use for overwintering, use to boost weak stores, etc. Give them room/ something to do. Be prepared to...