The GRIM test — a method for evaluating published research.

The GRIM test.

An Example of Granularity

How did this come about?

  • The GRIM test detects inconsistencies in the published means of small samples.
  • It is embarrassingly easy to understand and to run.

Using the GRIM test — What we did.

  1. sample size,
  2. the amount of decimal places reported in the mean, and
  3. the amount of subcomponents in each thing that’s measured (i.e. a scale with 7 items has more sub-components than a scale with 2 items).

1 in 2 papers that we checked reported at least one impossible value.

1 in 5 papers that we checked reported multiple impossible values.

What’s going on?

1. Us evaluating a mean incorrectly / making our own mistakes

2. Incorrect reporting of cell sizes

3. Bad reporting of composite measures

4. Typo

5. Not accounting for missing data

6. Fraud

Fraud.

The Real Problem

  • 2 authors, even though we confirmed their institutional emails were current, never replied at all to any email
  • 2 authors who were … let’s be charitable and say ‘hostile’ to the process
  • 2 authors who were perfectly happy to talk about the process but gently faded away when it was clear that we wanted to see their data
  • 2 authors who replied with identically worded refusals to share data, even though they seem to have no formal connection otherwise
  • 2 papers where one of the authors or associates is known to have committed research fraud previously…

Conclusion

I write about science. We can probably be friends.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

The Different Types of Plant Structures and Functions

Silver Nanoparticles in EU: a tough subject

A Letter to a philosopher of science from a scicommer

A generic drawing of a female figure and a generic drawing of a male figure standing together. The female figure has a square speech bubble that says: Look! Science Communication AND Philosophy of Science. The male figure looks away and has a round speech bubble that says: WHAT? Where?!!!!! The meme is meant to show that these two principles aren’t often seen as relevant to each other

Nikola Danaylov Virtual Keynote: How to Be a Futurist

The Tantalizing Tale of Taste

A One Hour Covid Test developed by an open-source team has advanced them to the semi-finals for…

Force diagram comment -Physics

The Pseudo-science of Human Evolution

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
James Heathers

James Heathers

I write about science. We can probably be friends.

More from Medium

The Machine Learning You did in High School

Some Tips to deal with Imbalanced Datasets

This is the imbalanced datasets where postive class is in miniority and negative class is in majority .

Outliers in data preprocessing

DEMYSTIFYING THE NAÏVE BAYES CLASSIFIER !!