The GRIM test — a method for evaluating published research.

The GRIM test.

The GRIM test is, above all, very simple.

An Example of Granularity

Let’s make a pretend sample of twelve undergraduates, with ages as follows:

How did this come about?

From analysing data that we were certain was fraudulent.

  • The GRIM test detects inconsistencies in the published means of small samples.
  • It is embarrassingly easy to understand and to run.

Using the GRIM test — What we did.

We took hundreds of papers recently published in psychology journals and GRIM-tested them.

  1. sample size,
  2. the amount of decimal places reported in the mean, and
  3. the amount of subcomponents in each thing that’s measured (i.e. a scale with 7 items has more sub-components than a scale with 2 items).

1 in 2 papers that we checked reported at least one impossible value.

1 in 5 papers that we checked reported multiple impossible values.

In papers with multiple impossible values, we asked the authors for the datasets they used. This was so we could a) see if the method worked and b) check the numbers up close.

What’s going on?

First of all, the GRIM test works very well, because we found an inconsistency in every dataset we received. These errors had a variety of sources:

1. Us evaluating a mean incorrectly / making our own mistakes

Make no mistake about it, we made some errors. This was a Herculean task, which involved hand-checking all the results from all 260 papers. Nick, whose focus and attention to detail is much better than mine, did most of the work. It was marvelous fun and by that I mean it was dreadful. We found two instances where we misunderstood the paper and checked something that turned out OK.

2. Incorrect reporting of cell sizes

This was very common — a paper would split a group of 40 people into two groups… and not tell you how big the groups were. You’d assume 20 each, right? Well, not so fast. Sometimes the groups were uneven (and this meant checking not just one mean, but all the possibilities) and everything appeared to be correct when we found consistent solutions with the published data. Other times, the cell sizes were wrong.

3. Bad reporting of composite measures

Sometimes, what we thought was a mistake might be the result of the items we scored having sub-items. For instance, if there was an impossible mean from a sample of n=20, but each person answered four questions to make up the mean, what appears to be a mistake might not be. Some papers left these details out.

4. Typo

Version control between authors, late nights, bad copy/paste job, spreadsheet mistake… it happens. We found Excel formulas that terminated in the wrong line, for instance. This was probably the major source of inconsistency overall.

5. Not accounting for missing data

Sometimes people in your study go missing, or drop out, or your equipment fails, or you spill coffee on your memory stick. Some papers report their overall sample sizes (how many people were enrolled in the study in the first place) but not how many people completed the study. Bit dodgy leaving these figures out — it never makes your paper look good to say “20 people started the study, but only 15 finished it” — but not a crime.

6. Fraud

… Yes, let’s talk about this.

Fraud.

Obviously, this is the big one.

The Real Problem

And while I’m being ominous here: we are far more concerned with the data we didn’t receive than the data we did receive.

  • 2 authors, even though we confirmed their institutional emails were current, never replied at all to any email
  • 2 authors who were … let’s be charitable and say ‘hostile’ to the process
  • 2 authors who were perfectly happy to talk about the process but gently faded away when it was clear that we wanted to see their data
  • 2 authors who replied with identically worded refusals to share data, even though they seem to have no formal connection otherwise
  • 2 papers where one of the authors or associates is known to have committed research fraud previously…

Conclusion

What happens from here will be interesting.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
James Heathers

James Heathers

I write about science. We can probably be friends.