Publication-status differences in priming effects disappear once correlated design variables are controlled.

Evidence

Since moderators of meta-analyses are not independent, we examined their interrelations. Inspection of the correlation matrix of study variables (see Table 4) showed that published studies were more likely to use self-reports, r =.49, (.20, .69), p = .002, less likely to use atheist or LGBT groups as targets, r = −.33, (−.57, −.04), p = .028, and more likely to be conducted outside the United States, r = .56, (.30, .74), p < .001. Therefore, we retested the effect of publication status, controlling for the three variables with which it was related. In this analysis, the difference between published and unpublished studies was no longer significant, b = . [−.03, .12], p = .28. — Annetta Snell and Miron Zuckerman and Bonnie M. Le, Does Religious Priming Induce Greater…, 84

Categorization

• XREF: Directly relevant to the reader’s question about self-report versus behavioral measures: published studies skew toward self-reports and away from atheist/LGBT outgroup targets, so apparent file-drawer effects may actually be design effects. — connects to: Comparing published and unpublished studies tests whether reporting bias inflates effects. — Thomas F. Pettigrew, Linda R. Tropp, A Meta-Analytic Test of Intergroup…, 109; Meta-analyses overestimate effect sizes because null results go unpublished. — Thomas F. Pettigrew, Linda R. Tropp, A Meta-Analytic Test of Intergroup…, 44
• DEEPTHINKING: A clean demonstration that an apparent publication-bias effect (published vs. unpublished differences) is actually carried by confounded study-design choices — a cautionary model for interpreting any meta-analytic moderator that is correlated with method.

Related

• The publication effect on priming prejudice results vanishes once confounding moderators are controlled (similarity 0.93)
• Comparing published and unpublished studies tests whether reporting bias inflates effects. (similarity 0.87)
• Religious priming’s apparent prejudice effect exists only in published studies. (similarity 0.87)
• Meta-analyses overestimate effect sizes because null results go unpublished. (similarity 0.87)
• Publication bias is unlikely to fully explain the religious priming effect. (similarity 0.85)
• Religious priming effects appear only in published, not unpublished, studies (similarity 0.84)
• Publication bias inflates the apparent effectiveness of prejudice-reduction programs (similarity 0.82)

Broader Connections

• Comparing published and unpublished studies tests whether reporting bias inflates effects. #expands (0.78) — Thomas F. Pettigrew, Linda R. Tropp, A Meta-Analytic Test of Intergroup…, 109
• Meta-analyses overestimate effect sizes because null results go unpublished. #qualifies (0.72) — Thomas F. Pettigrew, Linda R. Tropp, A Meta-Analytic Test of Intergroup…, 44