The Protraction of Retraction
A criminology paper has problems. What should be done?
So, what’s happening here? Short version which is approximately right:
- Group of authors of a Criminology paper receives information to suggest they published a paper which was full of errors.
- One author looks into this, and asks Serious Questions about his own paper. Answers are not forthcoming.
- He then produces this public excoriation of his own paper, which includes some bizarre stuff like ‘500 participants copied and pasted to produce 1000’ and ‘basic demographic information which can’t exist’. Fierce stuff.
- The whole kerfuffle makes it about the traps, and is reported in RetractionWatch.
- Journalist picks up on narrative, pretty common for ‘scandal in higher ed’ situations these days, and writes this.
That brings us to right now.
Statement #1
Sunday, I saw this.
And ‘imminently’ was yesterday. The statement is here.
The conflagration therein is focused on how the journal is handling this dispute. They are not considering a retraction at this juncture, rather the editors are strongly in favour of their existing model of handling disputes: the comment-and-reply approach.
The response they have written in justification is level-headed and straightforward, which is good.
I’ll quote the relevant paragraph in full.
As Co-Editors, we believe strongly that the comment-and-reply approach has several advantages [JH: they mean ‘over retraction’, presumably].
First, it allows — and encourages — readers to make their own judgments about the work in question. This form of active participation by readers is consistent with the nature of the scientific enterprise, and it allows for the ambiguities involved in almost all creative work.
Second, and relatedly, the approach allows for variations in the degree to which articles are problematic. It acknowledges a grey-scale world of more and less serious problems, and it does not impose black-and-white judgements that view articles as entirely correct or incorrect.
Third, the approach is fair to all parties. It gives authors and their critics equal opportunity to outline and defend their positions, and it avoids a situation in which the journal leaves authors essentially defenseless against attacks. Authors put themselves in a vulnerable position in publishing with a journal, and the comment-and-reply approach allows them to respond to critiques and correct mistakes if and when necessary.
Here’s my problem with this approach.
Journals and editors have a tremendous amount of vested authority. They are quite happy to be the arbiters of quality and interest in that they absolutely determine what they will and will not publish, what they deem interesting enough to review, how those reviews should be assessed to determine quality, and so on.
In other words, they claim the right to be perfectly qualified to make every single decision about every circumstance surrounding a paper up to and including publication, because they are the experts (and, I can assure you, they do not like having their expertise questioned).
Now, when there’s a big problem with something they have published, they still have the power. For the sake of convenience, let’s call me the interested reader cited above who is in the business of ‘making my own decision’.
The journal can marshall requests for the data which cannot be ignored — I can’t, people will ignore mine. (Note: people DO ignore mine).
The journal can access an array of experts on the content and statistics deployed — I can do this a degree, of course, but I do not have the same ability to compel assistance than the full editorial board.
The journal can ask pointed questions about the research process which, again, authors ignore at their peril — whereas I have all the authority of a school mackerel.
And in this situation, they somehow default onto letting the substantially less expert and less well resourced reader form their own opinions.
Presumably from (A) a commentary of limited length, linked somewhere obscure from the initial paper of concern, which in many cases is competing with (B) a response itself that is usually some combination of damage limitation, obscurantism, and mendacity.
Sounds like a good time to me!
This very measured approach makes a lot more sense within discussions of abstract theory, but almost none when it comes to concrete, non-abstract mathematical mistakes. If I claim that 1 + 1 = 257, and you publish it after it makes it through Deeply Infallible Peer Review, the need for ‘dialogue’ around my mistake is quite minimal — I have said something nonsensical or impossible. “[B]lack-and-white judgements that view articles as entirely correct or incorrect” are absolutely and completely possible, and happen all the time in a variety of fields.
Actually, it happens frequently in other cases a lot like this one: where the data in question is not only not released publicly but not even to a co-author, when a variety of the statistics are worrying inconsistent or impossible, and when the research process having been undertaken as reported seems, at minimum, extraordinarily unlikely.
Basically, when you put civility and open dialogue on such a pedestal that it overrides the need to pursue internal consistency or basic accuracy, your attitude is anti-scientific.
However, this statement is not a gainsaying of any possible retraction, but rather the description of a process which seems … well, highly protracted. What can be gleaned from this statement is that, and I’m only being somewhat presumptive here:
- this comment-and-reply business must first be established
- after that, a determination is made about whether or not it is necessary to publish the comment
- after publication, a determination is made about whether or not it is necessary to request the accompanying data or ask further questions
- after the data and errata are received (eventually), it will inspected, and then it will be reviewed by internal and external experts
- if that review is unfavourable, the subsequent situation after that require handling on an as-needs basis which does not explicitly discount retraction but after a ‘quasi-judicial process’
Basically, we can expect some kind of action on the document itself in approximately 2022, if everyone at every stage agrees there is a problem.
Which, historically speaking, is unlikely.
Now, let’s be fair: this whole business could be read as an attempt to process this rather difficult issue with fairness at every juncture.
But consider that the tremendous punitive weight of the retraction process on a complainant often involves (a) long periods of inaction during which ‘investigations’ are presumably proceeding, (b) stages where unwilling investigators or investigated authors can take their sweet time refusing to cooperate or failing to meet deadlines, sometimes for months or years, and subsequently (c) a tremendous ongoing social and professional pressure, sometimes punctuated by component time pressures to provide more information.
Of course, these inquiries often peter out and result in little action after everyone gets bored and moves on with their lives. Admit it, if you heard in 18 months time that this whole mess had been ‘resolved by discussion’, there’s no guarantee you’d remember a blessed word.
The only person who loses is the poor bastard who had the temerity to question the result in the first place. There is no place anywhere here, no consideration, for the position of the whistleblower. They have to live with whatever (undeserved) circumstances and reputation they have accrued in the meantime.
In any case: this statement certainly does represent a coherent position, and it could proceed fairly and reasonably quickly if successfully managed. It also could also be a smokescreen for total inaction, and window-dressing on the desire to do absolutely nothing about this whatsoever. Time will tell.
Statement #2
The above statement is, at least, both a clear statement of principles and outlines a plan congruent to those principles.
The below is… not.
This was leaked onto a sociology gossip board, also yesterday. It’s a statement from the editor of Criminology quoted in the Chron article. As far as I can tell, it has not been formally published or endorsed by the board of the journal.
And I can see why.
Upon editing, I redacted several uncomplimentary adjectives from this line. Unnecessary. I think it can speak for itself.
“In this document I discuss my reactions to “The Criminologist Accused of Cooking the Books,” by Tom Bartlett, in the September 24 issue of the Chronicle of Higher Education. I believe that the article distorts much of what I said in my interview with Mr. Bartlett, and that it holds Criminology up to unwarranted criticism.
Before I present my concerns, I would like to note three general points. First, the article is slanted to support a particular set of conclusions. It contains none of the “on-the-one-hand” versus “on-the-other-hand” presentation style that is usual in journalistic coverage of controversial topics. Instead, it assumes that a single position is correct, and it ignores opinions to the contrary. It appears that it intends to incite gullible readers to action, and judging from anonymous internet postings, it has succeeded in that goal. I think that the lack of objectivity is obvious, but I encourage anyone who doubts my claim to read the article closely.
Second, the article has an unpleasant, gossipy tone that stresses interpersonal conflicts. I wish to emphasize that my co-editors and I are concerned solely with the intellectual merits of matters involving the journal. We accordingly believe that personal issues should have absolutely no weight in our decisions. We find offensive the article’s insinuation that anything other than the quality of a submission might sway our judgments about it.
Third, I have talked to members of the press throughout my career, and Mr. Bartlett’s interviewing style baffled me. He was bullying, nasty, and aggressive, demanding justification for why Criminology did not promptly retract Eric Stewart’s article after “John Smith” and Justin Pickett complained about it. His questions had little to do with what the journal did in the case. Instead, they concentrated on why Criminology did not follow practices that he appeared to believe should be applied everywhere. He seemed more like a zealot promoting a religious principle than like a conscientious reporter.
At the beginning of the interview, I told Mr. Bartlett that our preferred approach to handling complaints followed a comment-and-reply model. This is a standard and well-worn method of dispute resolution in the social sciences, and it allows readers to form their own opinions about a controversy. To take only one example of its frequent use, the American Sociological Review has long employed this approach. Mr. Bartlett was dismissive of comments-and-replies, and he asked me whether we had requested Eric Stewart’s data and were considering retracting his article. I told him that we were not at the time either asking for the data or considering a retraction. I again explained the comment-and-reply approach. He was again unimpressed, sneering, and hostile, and he again repeated his questions. I again repeated my answers, and we continued cycling like this for an uncomfortably long period of time.
Because I described it so often, I was surprised to see that the Chronicle article did not mention our policies and procedures about handling disputed articles at all, and instead suggested that we were doing nothing. I think, after reading the article multiple times, that Mr. Bartlett assumed only actions related to a retraction qualified as a legitimate response to a complaint. Any other approach was muddle-headed and laughably “humane,” amounting to no action at all.
At some point during the interview, I acknowledged that circumstances did exist in which we would impose a retraction. I did this as much to change the subject as to present an alternative to comments-and-replies. I said that an imposed retraction was a final option, and that we would use it only after conducting a quasi-judicial review. The Chronicle article accurately reproduces my attempts to describe our views on the approach. It does so, however, in a context that emphasizes our reluctance to use this ultimate sanction, and it implies that I was otherwise indifferent to shoddy research. I stress as strongly as I possibly can that this is simply not true.
At another point during the interview, Mr. Bartlett asked why we were waiting for Eric Stewart to submit a report in defense of his work. He suggested that the complaints by “John Smith” and Justin Pickett were sufficiently compelling that they obviated the need for further investigation. I told him that fairness required that we allow both sides to present their positions. I said that John Smith and Justin Pickett’s complaints amounted to prosecutorial statements, written to make a case. I suggested that they were not necessarily objective, and that they could be colored by external factors. I pointed out that personal motivations, for example, might conceivably have influenced the complaints. The interview then veered off into irrelevant matters about personalities, and Mr. Bartlett would not allow a return to my emphasis on the desirability of balance. His article quoted my somewhat inflammatory speculations, said in the frustration of not being heard, and gave them significant attention. But it completely ignored my larger points.
I realized early in the questioning, after a few unproductive cycles of describing our approach, that Mr. Bartlett was not truly interested in listening to me. At that point, I probably should have ended the interview. I persisted nevertheless, because I could not understand his insistence that we immediately demand Eric Stewart’s data and retract his article. I assumed that I was not clearly describing our approach–although it seemed simple enough to me–so I continued despite the increasingly unpleasant atmosphere. Too late in the interview, I realized that I was losing connection to the issues, and that I was making statements Mr. Bartlett could use to defame the journal. I was not surprised that the article quoted these regrettable statements. It did surprise me, however, that it placed them in contexts very different from those in which I had made them.
I did not understand, and still do not understand, why Mr. Bartlett adopted such an aggressively hostile approach. He obviously had some type of agenda in mind, but it is not fully clear to me what that agenda might be. It seems to center on imposing severe mandatory punishments on those guilty — or even accused — of deficient research. Yet I fail to see how this advances science. I fear that if Criminology abandons its current approach and bows to bullying demands for a draconian retraction policy, the journal and the field will ultimately suffer for it. More generally, I think that in the long-term an inflexible retraction policy will harm any social science discipline that embraces it. Forcing all problems to fit a narrow and naively simple framework does not have a good history of success in the many settings where it has been tried.
Concluding, I had always assumed that the Chronicle of Higher Education was a thoroughly professional news organization, with a strong commitment to accurate reporting. That was not true here, and I am sorry the Chronicle apparently surrendered its integrity to force an unyielding ideological viewpoint onto a very human story.
David McDowall
Lead Co-Editor, Criminology
It’s worth going through some of this line by line, so we can see how the sausage is made.
“[The story in the Chron] contains none of the “on-the-one-hand” versus “on-the-other-hand” presentation style that is usual in journalistic coverage of controversial topics. Instead, it assumes that a single position is correct, and it ignores opinions to the contrary.”
Two problems here:
(1) false balance
The problem with strong inconsistencies in scientific documents—which the work in question very definitely contains — is that they can’t exist as described. They are unequivocally mistakes. Arithmetic is annoying like that*. So is duplication.
Thus to bemoan the fact a discussion about this paper doesn’t follow the time-honoured principle of ‘giving equal attention to both sides’ is to totally ignore the fact that quality and amount of evidence can exist. Bothsidesism in the face of very strong evidence is not a virtue.
If you want to know how false balance is a problem in wider discussion of serious issues, well, ask a climate scientist or a pediatrician handing out mandatory vaccinations.
(2) WHAT opinions to the contrary?
No defense is offered. What a position to be in as a journalist!
Journalist: here are some serious, carefully constructed, highly disturbing anomalies in a paper you published
Author: …
Journalist: go on, please talk to me
Author: …
Journalist: editor, any thoughts?
Editor: haven’t read it, author of complaint is probably a bad guy
Journalist: *publishes*
Editor: GOOD LORD MAN, HOW COULD YOU IGNORE OPINIONS TO THE CONTRARY??
No attempt has been made anywhere, by any other author or editor, to respond to any of the actual arguments provided in Pickett’s investigation of his own article.
I’m not entirely surprised. As someone who produces similar arguments, sometimes for money, I am extremely impressed with this document as a piece of criticism. It is clear, scrupulous, sophisticated, well-reported, and moderate. So in some respects, it feels unanswerable.
But, also, no-one tried.
This gets even more ridiculous, as if there is an investigation underway, the parties involved are most likely instructed to not talk to anyone about it. Totally normal for an internal investigation.
In short, a criticism for ‘ignoring opinions to the contrary’ is basically faulting the journalist here for not being Mystic Meg, and imagining what might be said in defense.
“He was bullying, nasty, and aggressive, demanding justification for why Criminology did not promptly retract Eric Stewart’s article after “John Smith” and Justin Pickett complained about it...”
“He was again unimpressed, sneering, and hostile, and he again repeated his questions.”
Now that’s a lot of adjectives.
I have been interviewed by Tom Bartlett three times, once in person. He struck me as a level-headed and deeply scrupulous man, with the threatening interpersonal demeanour of a bag of self-raising flour. Actually, I’m pretty sure all the above adjectives have been more frequently applied to me than they have to him.
I do remember him getting excited and pursuing some of the things I said doggedly (otherwise known as JOURNALISM) but I am also someone whose motives should be doggedly pursued. Also, that whole ‘agreeing to be interviewed’ thing.
Essentially, I absolutely do not recognise the domineering, arrogant bastard described in this statement who also is somehow a professional journalist.
Of course, I can’t defend his conduct here, because I wasn’t present. It’s all possible, and reasonably subjective in any case.
But I do find it an odd characterisation to suggest a journalist (and one, I might add, with substantial experience in writing about research integrity issues at a flagship publication focused on higher education, and almost certainly more experienced in talking to journal editors than any given random scicomm goon) would be so flagrantly antagonistic with the editor of a major journal.
“I suggested that they were not necessarily objective, and that they could be colored by external factors. I pointed out that personal motivations, for example, might conceivably have influenced the complaints. The interview then veered off into irrelevant matters about personalities…”
… Sorry, you were discussion potential personal motivations which were relevant, and then you veered off into the totally irrelevant subject of personalities?
“This [JH: comment and reply] is a standard and well-worn method of dispute resolution in the social sciences, and it allows readers to form their own opinions about a controversy.”
See above!
“Second, the article has an unpleasant, gossipy tone that stresses interpersonal conflicts. I wish to emphasize that my co-editors and I are concerned solely with the intellectual merits of matters involving the journal. We accordingly believe that personal issues should have absolutely no weight in our decisions.”
Two problems again:
(1) if you are solely concerned with the intellectual merits of the complaint, why wouldn’t you, uh, read the complaint in detail?***
He confirmed that he had seen the letter Pickett sent but says that he “didn’t read it in great depth.”
(2) the easiest way to avoid participating in an unpleasant, gossipy article is probably to avoid giving unpleasant, gossipy quotes for it:
“It seems to me that it’s pretty hostile for Justin to start making these claims,” McDowall says. Pickett says “negative things” about Stewart, according to McDowall, although he couldn’t recall specific statements. “I think he doesn’t like Eric personally and wants to ruin him and make him lose his job,” McDowall says.
It is also an incredibly bad look to go messenger-shooting and victim-blaming in academic processes. An experienced academic should have a very good idea of how this comes across, and the associations that it brings.
I was surprised to see that the Chronicle article did not mention our policies and procedures about handling disputed articles at all, and instead suggested that we were doing nothing.
Going from the previous statement from the full editorial board, it seems very obvious that something is being done, and it is indeed a failing of the Chron article if there’s a process at work (as Sisyphean as it might sound) which has gone unstated.
It suppose it would be more true to say ‘no public progress has been made on the issue since it was raised months ago, and very short, obvious questions which could be answered in a few idle hours by any honest researcher have not been asked’. Sadly, this is not at all atypical.
“Too late in the interview, I realized that I was losing connection to the issues, and that I was making statements Mr. Bartlett could use to defame the journal. I was not surprised that the article quoted these regrettable statements. It did surprise me, however, that it placed them in contexts very different from those in which I had made them.”
I would like to know what context “This is not the first time that papers were published in the journal that were complete gibberish” appears in which makes it more defensible.
“More generally, I think that in the long-term an inflexible retraction policy will harm any social science discipline that embraces it. Forcing all problems to fit a narrow and naively simple framework does not have a good history of success in the many settings where it has been tried.”
Who said anything about inflexibility? Retraction policies everywhere are notoriously flexible. They are fickle, subject to the personal whims of editors, frequently overruled by reticent publishers and editorial boards, inconsistently enforced, poorly deployed, often badly reported, and so on.
I would add that the vast majority of journals, and definitely all the high profile ones in STEM fields and the social sciences, retract papers for a variety of reasons. Somehow they have managed to do this for some decades without falling into Stalinist excess.
Conclusion
While I was writing this, I saw a RetractionWatch story which just broke. For once, it’s actually in my field. I’ve even seen the paper before.
The authors found an error buried deep within their own analysis while extending their existing work. What did they do? Chased it down, hit it with sticks, determined the extent of the mistake, talked to the editors, pulled their own paper.
The consequences of this, for me, are below:
THIS is science. This is not game-playing, dissembling, or haughty flim-flam about procedures which take 27 years to execute. This is the act of self-motivated, honest, intelligent people who are more interested in trying to solve a problem than collecting all the marbles.
So is this, from the other day:
Now, this might be overstating the case a bit (I fear deportation and penury a lot more than screwing up publicly, for instance) but the point is well made.
And the reaction in both cases has been hugely positive. Turns out people quite like consequential displays of intellectual humility, honesty, and rigour. Who’d have thought it?
Compare this environment to the previous, and criminology — the journal and the discipline — come off looking distinctly archaic, participants in a system of reputational fear and congenital paralysis, a world of worriers, chancers, endless bureaucracy, a discipline of a thousand tweed-bound Smaugs hoarding their Fancy Big Papers like gold coins, and a distinct lack of human progress.
Woe on their houses. I hope this happens again, and again, and again, until modernity takes hold. It’s not punitive, it’s progress.
* it’s borderline impossible to tell a huge and unwitting stuff-up apart from fabrication without examining the source material carefully, and as we can’t do that, there’s no reason to use the f-word. I’ve written about this extensively here. I don’t care about fraud or fabrication. I care about mistakes.
** note: this is why we spend so much of our time in error detection looking for anomalous numbers, rather than any other feature. If there is something wrong with an academic document, becoming mired in a discussion of angels and pinheads is about as much fun as a bracing morning swim through raw sewage.
*** if there’s one quote where I do question the context, it’s this one. Hopefully, this interview was conducted a sufficient amount of time ago that this has changed. However, no issue is taken with the characterisation… it would have been simple to say ‘well, after all this fuss, I’ve read it now!’
