The Buck Stops Nowhere
When research goes wrong, who’s responsible?
Earlier this year, five things happened in two weeks.
I wrote them all down in shorthand, then forgot about them. At least, for a while. The reason why is simple: I have a job and writing, well, writing takes time. The harvest of weeds and rocks and broken bottles that adulthood never promised but somehow delivered doesn’t do a bad job of crowding it out, either.
But I wasn’t exactly happy about what happened. These five little indignities stuck in my craw at the time, and I’ve been resolved for a while to put aside enough time to hit them with a chair leg. I was just waiting for the right late night to embrace the Unholy Systolic, and to march through red swirling mists screaming like the blinded Polyphemus and, coincidentally, clutching my forehead.
Also, one of the issues on this list was unresolved. It’s about to be resolved. So, the moon is high and the oars are in the rowlocks.
Anyway.
See if you can spot a common theme amongst the following incidents.
What Happened:
1. A reply from an editor.
Yes, Dr. Heathers, you ARE allowed to request data from people who publish in our journal, and yes, making the data available IS technically a stated condition of publication within this journal. You are technically correct when you outline these assertions, or, as you call them, ‘facts’.
However, I can’t actually compel the authors to do anything. Somehow we have no facility for that. I am bent over and brought low. Woe to me and mine.
Buuuuuuuuut, you know — nuts to you, McGillicuddy. Screw it.
In conclusion, go away.
P.S. Thank you so much for writing, and please be assured your concerns are taken seriously etc. etc. Please consider our journal for your future publications and inevitable slide into future senescence and wrack and ignominy and galloping heart disease, you mollusc.
(NOTE: paraphrasing somewhat.)
2. My friend Tim (amongst others) received the following anonymous cryptic email.
IF YOU THINK RESEARCHER X* DOES BAD RESEARCH, YOU SHOULD LOOK INTO THE WORK OF Y AND Z.
THEY ARE JUST AS BAD AS X! MAYBE WORSE!
IN FACT: I HEARD THEY BITE KITTENS!
I HEARD THEY LICK WOMEN’S SHOES ON THE TRAM!
DO NOT REPLY TO THIS EMAIL.
THIS MESSAGE WILL SELF-DESTRUCT IN TEN SECONDS.
#ominous ticking#
(*NOTE: X is a researcher we have previously criticized. And yes, this is also heavily paraphrased because I might as well enjoy myself.)
3. Some New Bad People (including yours truly) had to write a letter
The summary of that is: if you completely change a method’s section of a scientific paper to the extent that it irreparably alters the nature of the conclusions, you cannot explain it away. You cannot just issue a statement to say ‘yeah, well, the authors made a mistake, so let’s be understanding, and that’s the end of that’. This is awful.
Full text on The Colonel’s blog here.
4. We’re finally ready to publicly discuss some research which has *serious* problems. We’ve been pursuing this in private for NEARLY TWO YEARS.
What it says — I can’t tell you more, at least, not yet. But! Soon.
Long story short: Nick and myself (let’s be honest, mainly Nick) noticed some very serious irregularities in papers published by Researcher In Question (RIQ). We collected a dossier of several of these (individual papers, not mistakes), and presented it to Unnamed Official Body (UOB) to try to compel a response — entirely in private. No-one knows about this.
UOB have been lenient and understanding with both RIQ and with us. They have attempted to mediate a process that started slowly and became slower. And they were fobbed off, same as us. As I said, the process is now in its … I think in its 22nd month, without ever having really got off the ground.
We got two responses from RIQ. One was no good. The other was no good. So, neither UOB or us can compel a cogent response from RIQ. The time has come for sunlight.
You’ll see.
5. I found some really serious anomalies (by accident, actually) and I’m not quite sure what to do.
As per usual, by accident I stumbled on something… serious.
I can’t tell you more than this:
- plagiarism
- self-plagiarism
- heavy data reuse
- potential fabrication
- involves at least 7 papers, at least 2 severe / retraction-level
- field related to rather than in the social sciences
- one researcher, seven papers, four journals
So far.
Now what happens? I’m not sure who can be compelled to care. Each case is different — should we address the journal editors, author, general public…? This is serious, but I just don’t have time for this right now. The temptation is just to wash hands of it. I feel helpless, sometimes. I don’t wring my hands, but I can’t do this on my own. I’ve got work stacked like cordwood. I’ve got more unfinished manuscripts than teeth. And yet I am *compelled* to do something. Just.. you know… eventually.
How am I here again?
And What Of It All?
You’ve probably figured out what the consistent theme is here.
Hear that whirring? Hear that hum? Faintly, in the distance?
The industrial drone, behind the cars and the trucks and the quiet conversation and occasional bird brave enough to live in the city?
Is it a far off temple, where worshipers chant OM? Is it an underground substation, with warm, angry transformers? A generator, an air compressor? Is it the eighty-foot ghost of Otis Redding bumbling the rest of the song because he forgot the words?
Nah.
It’s the sound of everyone passing the buck.
The above, once more:
- an editor who has no power to enforce their stated ethical policy,
2. a bystander who sees problems in research, and writes to other people in secret, saying ‘can you look into this?’, and then disappears,
3. an editor who actively ignored the extent of obvious problems with a piece of published work,
4. a professional society who had an uncharacteristically valiant go, but in actual fact has no authority to compel answers about serious scientific problems,
5. New Bad Person (me!) unsure of how to deal with serious uncovered problems, because they don’t want it to become their personal responsibility, and because they’re convinced (rightly or wrongly) that they don’t have the bandwidth.
If there’s a theme, it’s “Sorry, there’s nothing I can do about this.”
The Broader Point:
Something that’s said to the New Bad People from time to time is: “hey, so you think you are the Data Police?”
The answer is always “no, but if you’ve got their phone number, I’d be DELIGHTED to turn some evidence over to them and get my f.ing nights back!”
Trust me, I’d be thrilled.
I don’t know if I’d make a good Data Police Officer, collecting evidence and building cases and grabbing collars, in fact I probably lack the disposition for the paperwork. But, goddamn it all red, I would make an excellent snitch. I’d be collecting grubby 20s like Bubbles from The Wire.
See, the police are a) a statutory body, and thereby b) compelled to investigate problems they find, especially if the allegations are serious enough. This is much better than our present ersatz version, with all of the responsibility and none of the formal authority! Please, someone, anyone, hold an election and commission a Data Sheriff. Like a Californian town in the Gold Rush — just pick the guy with the biggest moustache or something, and give him a tin star. They could have actual, real, real, actual authority.
(Not me, though, I look ridiculous in chaps.)
Of all the things to irk me, that researcher who wrote to say ‘hey, how about you investigate XYZ!’ is really beyond the pale. I needed a dose of nitroprusside to even sit down and think about what I’d write in reply:
Dear Mysterious Claimant,
(1) Could you literally be any less specific? How about [a] what problems you are referring to, and [b] where the problems are located in their entire corpus of someone else’s work?
(2) Could you provide any context as to how your claims arose?
(3) If these problems you allude to (without identifying) are serious enough to compel action , can I ask why you are reporting them anonymously to a grad student you found on the internet?
(4) Have you considered doing literally anything at all yourself about this problem? Would you consider shouldering a small portion of the responsibility for the task you’re handing out?
(5) Do you have access to anything that might improve the ability to investigate this claim, like, say, free money? Maybe a 28 hour day?
(6) Do you know how long this shit takes?
Thank you for writing. I appreciate your concerns, really — but I also regard you as the scientific equivalent of that person on the news who ‘conveniently forgets’ their baby in a KMart toilet.
Please attempt to be somewhat less than utterly, totally, and completely vague in future communications. Especially when trying to make work for other people.
P.S. Why did you do this?
Actually, that’s disingenuous. I know why.
I’ve mentioned previously the fact that methodological criticism and error detection is bad for the business model of producing scientific output, bad in a way that even the most vicious of fights between dueling ideas, fields, or research traditions is not. Proposing new ideas in the face of bad work is playing the game. Producing work in opposition to bad work is playing the game. Even having a vicious and protracted argument in a series of response papers is still playing the game.
But methodological criticism is not.
Criticism is sowing salt in the playing field. It is calling the game out publicly for causing CTE or having an entrenched drug problem. It is hiding the bases and stealing the puck. It is telling the players that the rules are broken and need recalibration, and that they should strike. It is destructive to business as usual — opinions differ markedly on the health and sanity of business as usual, of course.
But there’s also another, more selfish reason we don’t see this as important.
Imagine a normal CV for a research-only academic. It consists of the following achievements, stacked in approximate order of importance:
- grants won,
- grants won (yes, again… they’re everything),
- papers published (especially if they let you write more grants),
- conference presentations and keynotes delivered,
- editorships, academic ‘service’ (e.g. doing peer review), invited talks, teaching, outreach, and so on.
What you receive no formal credit for, at all, ever, is criticism. In the present academic environment, everything is measured … except this.
For anything else more consequential than blowing your nose, we get a gold star. How good is that paper? (journal ‘quality’, impact factor, Altmetric, etc.) How senior is that researcher? (K-index, etc.) How often are you cited? (various ways of counting exist). How much reviewing have you done? (Publons). What institute do you work at? (a constellation of reputation metrics). Have you given a lot of talks? (where, and how prestigious were the conferences/institutions where they were given?) Reputation and interest based metrics infiltrate everything.
But criticism isn’t measured, in fact, it is not even considered ‘service’, a catch-all term for unpaid yet necessary sideline tasks to academic life. It is not considered at all.
An additional perspective is also instructive. Imagine reading the following:
- “responsible for three corrections and two retractions of terrible work which wasted hundreds of thousands of $ / thousands of work hours”
- “hounded Journal XYZ into upholding their stated publication standards”
- “author of at least thirty angry letters to editors, resulting in etc. etc.”
Of course, it isn’t exactly easy to measure, but that is not the point here — the point is that the above is simply unthinkable for someone inside the academic tent. These sound like the career achievements of a curmudgeon, a thug or a crank. Even to me, these points, this reads as the brag sheet of a five-year-old boy who is proud of how many blocks he can kick over, wantonly destructive and oddly specific.
Even the above in its softest and most milquetoast academese version (“I am an active member of a community proudly committed to the improvement of standards etc. etc.”) will offer you very little to aid in The Scramble a.k.a. our headlong stumbling flight up a ziggurat made of rusty knives and emails which begin with the phrase “Thank you for application, but…” which provoke midday drinking.
In other words, the incentive to blurt out as much science as possible is presently coupled with the complete lack of incentives to curb the worst excesses within the blurting of others. Our anonymous researcher above sees no future in the task of unpacking what he/she sees as reams of obviously bad papers, cynically thrust into the world entirely for personal gain.
The public good, of course, is tremendous. It would be reasonable if that was a concern which mattered more to us, in our publicly-supported institutions where we endeavour to take public money and turn it into public knowledge.
A while ago, kids in town here were knocking around in shirts that said MAKE RACISTS AFRAID AGAIN. Right sentiment, but too much sugar in the tea for me. I’d settle for MAKE BAD SCIENTISTS HESITANT AGAIN.
Doesn’t make a snappy t-shirt, I guess. But as sure as God’s got sandals, it’s one way you can help from the comfort of your own couch.
If you can hold onto that buck long enough, that is.
Obligatory links: My Twertle, where I also have opinions, just shorter.
