But just how big should the effect be anyway? A key question for statistical inference

Abstract: A standard paradigm in statistical inference is to consider two interpretations of the results of an experiment: H0, in which there is no effect, and H1, in which there is some non-zero effect. However, a major stumbling block in this formulation is that if an effect is present, its magnitude is uncertain. I review how the different proposed solutions to this problem using significance testing and Bayesian hypothesis testing are also the source of critical flaws in these approaches. I then describe two situations in which the magnitude of a theoretically interesting effect size can be identified a priori: Replication attempts, in which the effect size can be inferred from the design of a previous study, and pre-registered studies, in which the effect size is based on theoretical and methodological considerations. When the magnitude of a theoretically interesting effect can be identified in this way, a comparison of the two interpretations is straightforward and efficient.

Visit Professor Dixon's website.