Mindless statistics

doi:10.1016/j.socec.2004.09.033

The Journal of Socio-Economics

Volume 33, Issue 5, November 2004, Pages 587-606

https://doi.org/10.1016/j.socec.2004.09.033 Get rights and content

Abstract

Statistical rituals largely eliminate statistical thinking in the social sciences. Rituals are indispensable for identification with social groups, but they should be the subject rather than the procedure of science. What I call the “null ritual” consists of three steps: (1) set up a statistical null hypothesis, but do not specify your own hypothesis nor any alternative hypothesis, (2) use the 5% significance level for rejecting the null and accepting your hypothesis, and (3) always perform this procedure. I report evidence of the resulting collective confusion and fears about sanctions on the part of students and teachers, researchers and editors, as well as textbook writers.

Section snippets

The null ritual

Textbooks and curricula in psychology almost never teach the statistical toolbox, which contains tools such as descriptive statistics, Tukey's exploratory methods, Bayesian statistics, Neyman–Pearson decision theory and Wald's sequential analysis. Knowing the contents of a toolbox, of course, requires statistical thinking, that is, the art of choosing a proper tool for a given problem. Instead, one single procedure that I call the “null ritual” tends to be featured in texts and practiced by

What Fisher and Neyman–Pearson actually proposed

In discussions about the pros and cons of significance testing in the social sciences, it is commonly overlooked (by both sides) that the ritual is not even part of statistics proper. So let us see what Fisher and Neyman–Pearson actually proposed. The logic of Fisher's (1955, 1956) null hypothesis testing can be summarized in three steps:

Feelings of guilt

Let me introduce Dr. Publish-Perish. He is the average researcher, a devoted consumer of statistical packages. His superego tells him that he ought to set the level of significance before an experiment is performed. A level of 1% would be impressive, wouldn’t it? Yes, but … He fears that the p-value calculated from the data could turn out slightly higher. What if it were 1.1%? Then he would have to report a nonsignificant result. He does not want to take that risk. How about setting the level

Collective illusions

Rituals call for cognitive illusions. Their function is to make the final product, a significant result, appear highly informative, and thereby justify the ritual. Try to answer the following question (Oakes, 1986, Haller and Krauss, 2002):

Suppose you have a treatment that you suspect may alter performance on a certain task. You compare the means of your control and experimental groups (say 20 subjects in each sample). Further, suppose you use a simple independent means t-test and your result

An editor with guts

Everyone seems to have an answer to this question: Who is to blame for the null ritual? Always someone else. A smart graduate student told me that he did not want problems with his thesis advisor. When he finally got his Ph.D. and a post-doc, his concern was to get a real job. Soon he was an assistant professor at a respected university, but he still felt he could not afford statistical thinking because he needed to publish quickly to get tenure. The editors required the ritual, he apologized,

The superego, the ego, and the id

Why do intelligent people engage in statistical rituals rather than in statistical thinking? Every person of average intelligence can understand that p(D|H) is not the same as p(H|D). That this insight fades away when it comes to hypothesis testing suggests that the cause is not intellectual but social and emotional. Here is a hypothesis (Acree, 1978, Gigerenzer, 1993): The conflict between statisticians, both suppressed by and inherent in the textbooks, has become internalized in the minds of

Meehl's conjecture

Paul Meehl, a brilliant clinical psychologist with a broad interest in the philosophy of science, was one of those who blamed Fisher for the decline of statistical thinking in psychology. “Sir Ronald has befuddled us, mesmerized us, and led us down the primrose path. I believe the almost universal reliance on merely refuting the null hypothesis … is a terrible mistake, is basically unsound, poor scientific strategy, and one of the worst things that ever happened in the history of psychology” (

Feynman's conjecture

The routine reliance on the null ritual discourages not only statistical thinking but also theoretical thinking. One does not need to specify one's hypothesis, nor any challenging alternative hypothesis. There is no premium on “bold” hypotheses, in the sense of Karl Popper or Bayesian model comparison (MacKay, 1995). In many experimental papers in social and cognitive psychology, there is no theory in shooting distance, but only surrogates such as redescription of the results (Gigerenzer, 2000,

The dawn of statistical thinking

Rituals seem to be indispensable for the self-definition of social groups and for transitions in life, and there is nothing wrong with them. However, they should be the subject rather than the procedure of social sciences. Elements of social rituals include (i) the repetition of the same action, (ii) a focus on special numbers or colors, (iii) fears about serious sanctions for rule violations, and (iv) wishful thinking and delusions that virtually eliminate critical thinking (Dulaney and Fiske,

References (64)

N.G. Waller
The fallacy of the null hypothesis in soft psychology
Applied and Preventive Psychology
(2004)
Acree, M.C., 1978. Theories of Statistical Inference in Psychological Research: A Historicocritical Study....
American Psychological Association, 1974. Publication Manual, 2nd ed., 3rd ed., 1983; 4th ed., 1994; 5th ed., 2001....
A. Anastasi
Differential psychology
(1958)
D.R. Anderson et al.
Null hypothesis testing: problems, prevalence, and an alternative
Journal of Wildlife Management
(2000)
D. Bakan
The test of significance in psychological research
Psychological Bulletin
(1966)
S.L. Chow
Précis of “Statistical significance: rationale, validity, and utility”
Behavioral and Brain Sciences
(1998)
J. Cohen
The statistical power of abnormal-social psychological research: a review
Journal of Abnormal and Social Psychology
(1962)
J. Cohen
Statistical power analysis for the behavioral sciences
(1988)
Czerlinski, J., Gigerenzer, G., Goldstein, D.G., 1999. How good are simple heuristics? In: Gigerenzer, G., Todd, P.M.,...

S. Dulaney et al.

Cultural rituals and obsessive-compulsive disorder: is there a common psychological mechanism?

Ethos

(1994)

R. Falk et al.

Significance tests die hard

Theory and Psychology

(1995)

L. Ferguson

Statistical Analysis in Psychology and Education

(1959)

R. Feynman

The Meaning of it All: Thoughts of a Citizen-Scientist

(1998)

F. Fidler

The fifth edition of the APA Publication Manual: why its statistics recommendations are so controversial

Educational and Psychological Measurement

(2002)

Finch Cumming, G., Williams, J., Palmer, L., Griffith, E., Alders, C., et al. 2004. Reform of statistical inference in...

Fisher, R.A., 1935. The design of experiments, 5th ed., 1951; 7th ed., 1960; 8th ed., 1966. Oliver & Boyd,...

R.A. Fisher

Statistical methods and scientific induction

Journal of the Royal Statistical Society (B)

(1955)

R.A. Fisher

Statistical Methods and Scientific Inference

(1956)

M.A. Gernsbacher

Editorial comment

Memory and Cognition

(1998)

R.J. Gerrig et al.

Psychology and Life

(2002)

G. Gigerenzer

Probabilistic thinking and the fight against subjectivity

G. Gigerenzer

The superego, the ego, and the id in statistical reasoning

G. Gigerenzer

Adaptive Thinking: Rationality in the Real World

(2000)

Gigerenzer, G., 2002. Calculated Risks: How to Know When Numbers Deceive You. Simon & Schuster, New York (UK edition:...

G. Gigerenzer et al.

The null ritual: What you always wanted to know about null hypothesis testing but were afraid to ask

G. Gigerenzer et al.

Cognition as Intuitive Statistics

(1987)

G. Gigerenzer et al.

The Empire of Chance. How Probability Changed Science and Every Day Life

(1989)

Gigerenzer, G., Todd, P.M., The ABC Research Group, 1999. Simple Heuristics that Make Us Smart. Oxford University...

Guilford, J.P., 1942. Fundamental Statistics in Psychology and Education, 3rd ed., 1956; 6th ed., 1978 (with Fruchter,...

Haller, H., Krauss, S., 2002. Misinterpretations of significance: a problem students share with their teachers? Methods...

U. Hoffrage et al.

Hindsight bias: a by-product of knowledge updating?

Journal of Experimental Psychology: Learning, Memory, and Cognition

(2000)

Cited by (623)

Inconsistent multiple testing corrections: The fallacy of using family-based error rates to make inferences about individual hypotheses
2024, Methods in Psychology
During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union alternative hypothesis (e.g., “H_1,1 or H_1,2”). However, in some cases, they do not make this inference. Instead, they make separate inferences about each of the individual hypotheses that comprise the joint hypothesis (e.g., H_1,1 and H_1,2). For example, a researcher might use a Bonferroni correction to adjust their alpha level from the conventional level of 0.050 to 0.025 when testing H_1,1 and H_1,2, find a significant result for H_1,1 (p < 0.025) and not for H_1,2 (p > 0.025), and so claim support for H_1,1 and not for H_1,2. However, these separate individual inferences do not require an alpha adjustment. Only a statistical inference about the union alternative hypothesis “H_1,1 or H_1,2” requires an alpha adjustment because it is based on “at least one” significant result among the two tests, and so it refers to the familywise error rate. Hence, an inconsistent correction occurs when a researcher corrects their alpha level during multiple testing but does not make an inference about a union alternative hypothesis. In the present article, I discuss this inconsistent correction problem, including its reduction in statistical power for tests of individual hypotheses and its potential causes vis-à-vis error rate confusions and the alpha adjustment ritual. I also provide three illustrations of inconsistent corrections from recent psychology studies. I conclude that inconsistent corrections represent a symptom of statisticism, and I call for a more nuanced inference-based approach to multiple testing corrections.
Measures, constructs, and constructing measurement: Reply to comments on “Can we really ‘read’ art to see the changing brain?”
2024, Physics of Life Reviews
Statistical approach for radioactivity detection: A brief review
2024, Journal of Environmental Radioactivity
Radioactivity detection is a major research and development priority for many practical applications. Amongst the various technical challenges in this field is the need to carry out accurate low-level radioactivity measurements in the presence of a large fluctuations in the natural radiation background, while reducing the false alarm rates. The task becomes even more harder with high detection limits under low signal-to-background ratios. A detection method based on the statistical inference, following either a frequentist or a Bayesian paradigm, adopted to overcome these challenges as well as to ensure a reliable and accurate diagnosis with a competitive tradeoff between sensitivity, specificity and response time. With this respect, several research studies, addressing a range of applications from decommissioning and dismantling to homeland security, have been proposed. Our main goal in this paper is to present a succinct survey of these studies based on a frequentist and Bayesian approaches used to decision-making, uncertainty and risk evaluation, in the context of radioactive detection. In this prospect, a theoretical background of statistical frequentist and Bayesian inferences was presented. Then, a comparative study of both approaches was performed to determine the optimal approach in regards to accuracy and pros/cons. A case of study for low-level radioactivity detection in nuclear decommissioning operations was provided to validate the optimal approach. Results proved the efficiency and usefulness of Bayesian approach against frequentist one with respect to the most challenging scenarios in radiation detection applications.
When expert predictions fail
2024, Trends in Cognitive Sciences
We examine the opportunities and challenges of expert judgment in the social sciences, scrutinizing the way social scientists make predictions. While social scientists show above-chance accuracy in predicting laboratory-based phenomena, they often struggle to predict real-world societal changes. We argue that most causal models used in social sciences are oversimplified, confuse levels of analysis to which a model applies, misalign the nature of the model with the nature of the phenomena, and fail to consider factors beyond the scientist’s pet theory. Taking cues from physical sciences and meteorology, we advocate an approach that integrates broad foundational models with context-specific time series data. We call for a shift in the social sciences towards more precise, daring predictions and greater intellectual humility.
The roles, challenges, and merits of the p value
2023, Patterns
Since the 18th century, the p value has been an important part of hypothesis-based scientific investigation. As statistical and data science engines accelerate, questions emerge: to what extent are scientific discoveries based on p values reliable and reproducible? Should one adjust the significance level or find alternatives for the p value? Inspired by these questions and everlasting attempts to address them, here, we provide a systematic examination of the p value from its roles and merits to its misuses and misinterpretations. For the latter, we summarize modest recommendations to handle them. In parallel, we present the Bayesian alternatives for seeking evidence and discuss the pooling of p values from multiple studies and datasets. Overall, we argue that the p value and hypothesis testing form a useful probabilistic decision-making mechanism, facilitating causal inference, feature selection, and predictive modeling, but that the interpretation of the p value must be contextual, considering the scientific question, experimental design, and statistical principles.
When perceived informativity is not enough. How teachers perceive and interpret statistical results of educational research
2023, Teaching and Teacher Education
Teachers are encouraged to inform their professional actions by evidence. Consequently, it is important to know how they perceive and understand statistical results. In a representative sample of 414 teachers, we presented typical descriptions of inferential statistics and effect sizes. Results suggest that teachers perceive the standard significance statement, Cohen's U₃, and a combined description of inferential statistic and effect size as most informative. Overall, however, they do not disentangle inferential statistics and effect sizes correctly. Designing informative and easy-to-understand science communication appears to be challenging. Hence, we suggest to also foster teachers' research literacy to improve their evidence-informed actions.

View all citing articles on Scopus

View full text

Mindless statistics

Abstract

Section snippets

The null ritual

What Fisher and Neyman–Pearson actually proposed

Feelings of guilt

Collective illusions

An editor with guts

The superego, the ego, and the id

Meehl's conjecture

Feynman's conjecture

The dawn of statistical thinking

Applied and Preventive Psychology

Differential psychology

Null hypothesis testing: problems, prevalence, and an alternative

Journal of Wildlife Management

The test of significance in psychological research

Psychological Bulletin

Précis of “Statistical significance: rationale, validity, and utility”

Behavioral and Brain Sciences

The statistical power of abnormal-social psychological research: a review

Journal of Abnormal and Social Psychology

Statistical power analysis for the behavioral sciences

Cultural rituals and obsessive-compulsive disorder: is there a common psychological mechanism?

Ethos

Significance tests die hard

Theory and Psychology

Statistical Analysis in Psychology and Education

The Meaning of it All: Thoughts of a Citizen-Scientist

The fifth edition of the APA Publication Manual: why its statistics recommendations are so controversial

Educational and Psychological Measurement

Statistical methods and scientific induction

Journal of the Royal Statistical Society (B)

Statistical Methods and Scientific Inference

Editorial comment

Memory and Cognition

Psychology and Life

Probabilistic thinking and the fight against subjectivity

The superego, the ego, and the id in statistical reasoning

Adaptive Thinking: Rationality in the Real World

The null ritual: What you always wanted to know about null hypothesis testing but were afraid to ask

Cognition as Intuitive Statistics

The Empire of Chance. How Probability Changed Science and Every Day Life

Hindsight bias: a by-product of knowledge updating?

Journal of Experimental Psychology: Learning, Memory, and Cognition