Some of you may recall the events of late 2012 / early 2013, in which an anonymous blog I was running to highlight data problems in the life sciences literature, was rather unceremoniously shuttered by legal threats.
In late 2013, to try and salvage something of lasting benefit to the scientific community out of these unfortunate events, I conducted a post-hoc’ study to see what happened to the 274 papers I had blogged about – i.e., were they corrected or retracted from journals? As a control group for the study I used a set of 223 papers I had acquired during the course of running the blog, but not yet got around to writing about.
The results, which were published in April 2014 in this PeerJ paper, showed that papers blogged about were acted upon at a significantly greater rate than those whose data problems were kept private. Specifically, the blogged papers were retracted 6.5-fold more, and corrected 7.5-fold more. Thus, the overall conclusion was that exposure of problem data in a public forum had a large impact on whether the journals/institutions/authors actually chose to do anything about it.
So, why the update?
One of my lingering doubts about this study, was that the private papers were received across an ever-so-slightly later time window than the blogged ones. Specifically, the blogged papers were received June ’12 – December ’12, while the private papers were received November ’12 – January ’13. Even though the data for the study were collected right up until final submission of the paper in February 2014, there was always the nagging possibility that the private papers may have been acted on less, because less time had elapsed for them. Given sufficient time, would the private papers catch up with the blogged ones?
I think it can now be argued that sufficient time has elapsed. In addition, the dramatic rise of PubPeer, PubMedCommons, and post-publication peer review in general, means there have been multiple other opportunities for the people who sent me these papers to criticize them on-line by now. So, knowing (as I do) that several of the private papers have indeed appeared on PubPeer, one might predict they’d catch up with the blogged ones?
However, it turns out that my nagging self-doubt was ill-founded. As of today (January 2016) the blogged papers have continued to be retracted and corrected at a much greater level than the private ones!
Specifically, the blogged papers have racked up another 14 errata and 10 retractions in the intervening 2 years, bringing their totals to 61 errata and and 26 retractions (an almost 32% overall rate of action). In contrast during the same period the private papers have garnered another 2 errata and 1 retraction, for respective totals of 7 and 3, or a 4.5% overall action rate.
In other words, over 3 years since the shuttering of the blog, the blogged papers continue to be acted on at a compounded rate that is 7-fold greater than the private papers.
So, the previous advantage (?) conferred to a paper by having been blogged about, seems to have held up over the intervening period. The private papers didn’t catch up, and while I’ll hesitate to predict they never will, the overall pattern doesn’t show any signs of changing. Bottom line – exposure works, and perhaps even some kinds of exposure (the snarky blog variety) work better than others.
FYI, I’ll be speaking about these data at the upcoming “Gaming Metrics” conference at UC Davis, next month.