This is my question for PLoS Biology today.  You may recall some time ago I spoke out regarding what appeared to be data of questionable integrity in paper at the journal. Last month I provided a follow-up to coincide with the 6 month anniversary of the paper’s publication, highlighting a complete lack of action taken in response to my findings. Well today, I got the following email from the editor…

Dear Dr. Brookes,
I am writing to let you know that we have issued a formal correction for the manuscript entitled “Effects of Resveratrol and SIRT1 on PGC-1α Activity and Mitochondrial Biogenesis: A Reevaluation” by corresponding author John Holloszy, published in PLOS Biology. The authors inadvertently used the wrong blots in three of the main figures resulting in duplications. You can follow this link to see the correction: http://www.plosbiology.org/annotation/listThread.action?root%3D78137
Thank you for contacting us regarding these issues.
Sincerely

The correction itself reads as follows:

The authors inadvertently used the wrong blots in three of the main figures resulting in duplications. The blots in question are PGC-1a and LCAD in Figure 1A, Cyto C in Figure 4C, and COXIV in Figure 6B. The authors and their institution have confirmed that these errors do not affect the interpretation of the results or conclusions of the paper. The authors apologize for any confusion caused by these mistakes and thank the readers for alerting them to these issues.
Please view the correct Figure 1 here:
http://www.plosbiology.or…
Please view the correct Figure 4 here:
http://www.plosbiology.or…
Please view the correct Figure 6 here:
http://www.plosbiology.or…

Ignoring the fact that the correction was issued a week ago, and they only just saw fit to tell me about it, I guess that clears everything up then!

Except…
(1) The problem with Figure 4A was not duplication, but splicing. Some panels of a blot were spliced (in one case, every lane of the blot) but the supposed loading controls were not spliced.  As such, these controls could not possibly have arisen from the same gel or membrane and are therefore not proper loading controls.  This issue is completely un-resolved by the correction.

(2) The correction claims that the authors “inadvertently used the wrong blots in three of the main figures”.  To me, “wrong blot” implies the whole blot. Not part of a blot. Not one lane of a blot. The whole thing. In the case of Figure 6 this may be an adequate explanation – the entire blot (all 4 lanes) was wrong, so the whole thing has been replaced in the correction. BUT, in the case of Figure 4C, the substitution was not a whole blot, it was a single lane of a blot. How exactly does one “inadvertently” use the wrong band in a blot?

Just to be clear, the phrase “inadvertently used the wrong blots” is completely inadequate as an explanation for what actually gave rise to the figure – duplication of a single band between blots via un-disclosed splicing.

In the case of the LCAD blot in Figure 1, again it was not simple duplication of an intact blot – the lanes were shifted over by one position. Is “used the wrong blots” an adequate explanation, for the practice of using the same blot panel and shifting the lanes over by one position (not to mention adjusting the saturation) to represent different experiments? Is “duplication” an adequate explanation for what happened here, given the manipulation that took place to give rise to the different images.

(3) The corrected images are interesting to say the least.  Each image is presented as a TIF, which is not particularly problematic, until you look at the horizontal/vertical alignment.  To do this, open a Figure (say Fig. 4), hold down the CTRL key and roll the mouse wheel to magnify the image.  Then scroll vertically so one of the features which should be horizontal (e.g. a graph x-axis) is aligned with the upper or lower boundary of your computer window.  Notice anything funny?

That’s right- these images are not aligned, as they would be if they’d been generated from a software package such as PowerPoint or Illustrator. I did this for the graph at the bottom of Fig. 4, to demonstrate…

…but it’s the same for every part of the Figure, including the bottom edges of all the blot images. How did these figures come to be rotated ever so slightly?  My speculation is that they were scanned in from a paper copy at some point, and someone mis-aligned the paper in the scanner.

So here’s the problem… Why, when the integrity of these data was being questioned, would the authors choose to use such an arcane method of figure preparation which raises more questions than it answers?

Why did it take 7 months for the journal to do anything about this? Do they honestly expect their readers to accept such a totally inadequate explanation from these authors? Do they really expect us to buy that this “does not affect the interpretation of the results or conclusions of the paper”? Do they think I was born yesterday?

Naturally, a response email calling out the above issues has already been sent, and posts to PubPeer, PubMed Commons and the PLoS Biology page itself, are in progress.

This is a long read, so here’s the short version: In 2012, some problems came to light in three papers with a common author. One journal retracted the paper, albeit after much effort and with an opaque retraction notice. Another acknowledged the problems but refused to act and is now incommunicado. At the third journal there is reason to suspect an investigation may have been compromised by conflict-of-interest. Combined, these events do not speak highly to the manner in which journals handle these affairs. What’s more COPE guidelines appear to encourage such practices. Oh, and just to complicate things, the author of the papers threatened to sue me. There is no happy ending.

Disclaimer and motives: I can’t get into the potential reasons why the data in these papers appear the way they do. All I can do is report on what’s out there in the public domain, and offer my own (first amendment protected) opinion about why these images appear odd to me. It should also be clear that in discussing these papers, I’m not drawing any parallels to online discussions that may have happened about them in the past. The opinions expressed here are my own (***see also bottom of page).

So why do this?  Well, I don’t like mistakes, and assuming everything I’m about to show you is the result of honest mistakes, that means: (i) journals are not doing their jobs as gatekeepers of reliable scientific information, (ii) people are not proof-reading their manuscripts before submission, and (iii) poor data management is being rewarded with grants, publications and career advancement. Because success in academic science is a zero sum game, when people who make mistakes get ahead, others are deprived of success. That’s not fair.

How it started: Alleged problems in the data of three papers from Gizem Donmez at Tufts University came to light in fall 2012:

J. Neurosci. 2012;32;124-32; PMID22219275
J. Biol. Chem. 2012;287;32307-32311; PMID22898818
Cell 2010;142;320-332; PMID20655472

The problems appeared to involve undisclosed splicing together of western blot images, and in some cases apparent re-use of western blots (or portions thereof) across different figures. Detailed descriptions are available at PubPeer (J Neurosci / J Biol Chem / Cell) for those interested. I have been involved for over a year in trying to ensure the journals deal with these alleged problems appropriately. What follows is a summary of what happened in each case. (Note – some email content is redacted for brevity, but full transcripts can be provided on request)…

What happened at J. Neurosci. 7 months after my initial contact, Editor-in-Chief John Maunsell responded:

The SfN requested institutional investigations […] institutions produced detailed reports that included the original images used to construct the figures. The images clearly documented that the figure elements that appeared to be replicates in the article in J. Neurosci. were in fact from different gels. Because there was no evidence that any results had been misrepresented, the SfN has dismissed the case. We consider the matter closed.

I protested, asking why the issue of undisclosed splicing was not resolved. I also questioned whether the journal had actually seen the originals or was simply relying on the institutional report? This was the response:

Your concerns about the spliced gels did not escape our attention.  We did not take action for two reasons. First, the image manipulation policy of J. Neurosci. was only instituted in December 2012, after this article was published. While it is regrettable that we did not have a more rigorous explicit policy in place sooner, we cannot impose retroactive policies on published articles. Second, while the spicing misrepresented details of the experiment design, the institutional investigations left us with no reason to believe they misrepresented any of the scientific findings of the study. While there are good reasons why authors should not manipulate images in this way, the figures appear to faithfully reproduce the results obtained, and would not impede anyone from replicating the author’s results. Because none of the scientific finding were represented, a correction would provide no value for readers of the article.

Refusing to retroactively apply a policy that every other journal had in place years ago, seems rather odd, as does the assertion that a correction would provide no value. Anyway, I let it go for a while then wrote back again in fall 2013, notifying that the JBC paper had been retracted (see below), and the lead author had threatened to sue me. The journal responded:

We are unprepared to reopen this case.  The events surrounding other publications and your interactions with Dr. Donmez are immaterial for this matter. Your assertion […] is misguided. I acknowledged these images look very similar, but the institutional investigation included examination of the original, high-resolution, uncropped images, and the material presented to us** showed the similarity was a remarkable coincidence.  I understand that you and others will find that surprising, but there is no arguing with the originals. We consider this matter closed.

** It is unclear if the journal has actually seen the originals, or is merely relying on what has been been presented to them by the institution. So I wrote back again, this time documenting a further analysis of the data, including the following images:

While it’s technically feasible those images originated from different experiments, it would represent a remarkable coincidence!  Maybe I should go buy lottery tickets right now? Anyway, it doesn’t matter because the journal didn’t respond to my email. Four months later they’re still ignoring me.

What happened at J. Biol. Chem.  JBC‘s dedicated ethics officer Patricia Valdez has been a model citizen for interactions with readers and forwarding the journal’s agenda of increased transparency. That being said, things got interesting in May 2013 when JBC published a correction to the questioned paper. The correction acknowledged that some western blots in the paper were subjected to splicing, and replacement figures were provided with splicing seams clearly indicated by solid lines. The correction stated that the splicing “in no way affects the conclusions of the paper or the original interpretation of the results”. If only it were that simple…

Straight away it came to my attention that, while acknowledging the splicing, the correction failed to address the issue that images on each side of the splicing seams appeared similar. So, I wrote to JBC with a detailed analysis of the corrected images, including the following:

After walking Ms. Valdez through the images in a ‘phone conversation, she agreed that things deserved another look. She also indicated by email that the initial decision to correct was based on an institutional report (sound familiar?). After several follow-up communications the journal issued a retraction notice on July 24th 2013. Here’s the full text of that notice:

This article has been withdrawn by the authors

Move along, nothing to see here! I won’t speculate on the reasons why JBC broke protocol and permitted an opaque notice. However, I will note that this occured on the same day as Dr. Donmez threatened to sue me. In addition Diane Souvaine (Dean at Tufts) stopped answering my emails around the same time.

What happened at Cell. The journal didn’t respond to my original contact in fall 2012, so I tried again in fall 2013 (yes thank-you I have a day job that keeps me busy, hence the delay). In my email I emphasized the importance of avoiding conflict-of-interest, because the senior author on the Cell paper (Leonard Guarente at MIT) sits on the editorial board of the journal. A week later, Editor-in-Chief Emilie Marcus responded that they’d look into it. Nothing was heard for 2 months, then last week I received the following email:

Dear Dr. Brookes,
Thank you for your e-mails. In addition to having been informed of the results of the institutional investigation, we have also examined the implicated figure panels editorially. Despite some apparent superficial similarities, upon extensive examination we were unable to find any compelling evidence for manipulation or duplication in those panels and therefore are not taking any further action at this time.
Best wishes,
Sri Devi Narasimhan, PhD, Scientific Editor, Cell

Having not seen the name before, I looked into the background of Dr. Narasimhan. It turns out she came from the lab’ of Heidi Tissenbaum at UMass. Guess where Dr. Tissenbaum did her post-doc! Can you say conflict-of-interest?

Despite being specifically warned about conflict, Cell put someone who is a scientific descendent of the lead author in charge of the investigation. The response also offered no details on the tools used to reach these conclusions, and no mention whether original images were requested. Furthermore, the response fails to address undisclosed splicing.

I responded, raising these issues and CC’ing the Committee on Publication Ethics (COPE). Despite additional ‘phone calls and even leaving my cellphone number, a week later I’ve received no response from Cell. So, in the interests of completeness, here’s another look at the data, plus a few more apparent problems I found along the way…

‘

Cell hasn’t seen this re-analysis yet, so there’s no word on whether they’ll re-open this case after reading the above. I’ll update if that changes.

What does (the) COPE say? I’m generally a fan of the Committee on Publication Ethics, although they don’t seem to have strong enough teeth to get journals to behave. In response to being CC’ed on my email to Cell, they replied:

Thank you for your email and telephone call […] COPE cannot investigate individual cases but we do hear concerns about member journals. More information and how to follow this process can be found here.

At the link, I learned that official complaints must state which part of the COPE code of conduct has been breached. In that code, editors are advised to “inform readers about steps taken to ensure submissions from members of the […] editorial board receive an objective and unbiased evaluation”. The Cell case appears to be a breach of code, but requiring a formal complaint to be filed places an unnecessary burden on the reader (me). Why cant COPE simply act on the evidence they’ve already received?

Furthermore, COPE guidelines for editors to handle such issues recommend deference to the authors’ institution for a full investigation. If the institution says OK, the journal should run with it. This policy ignores any potential conflict-of-interest at the institution itself (e.g., indirect costs on faculty grants), which to me seems like a rather large hole in the system… Let’s put people who stand to lose big heaps of money in charge, and hope they do the right thing.

What about the legal issue, and what next? In dealing with these papers and the ensuing legal issues, I’ve spent a lot of my own money. That’s money no longer in my kids’ college savings account. I also seriously doubt I’ll be reimbursed by the journals for the significant time I spent doing image analysis – analysis which they should have done during peer review.

There is no happy ending – just a refusal to act, an opaque retraction and a conflicted investigation. Meanwhile, an extremely talented junior investigator from a world-class institution continues to secure high profile grants and, with help from a good lawyer and the powerful Harvard/MIT/Tufts network, is all set for a stellar career. Let’s hope nobody looks at her earlier papers.

_______________

*** If you think anything written above is not a fair representation of the truth, I am open to discussion by email. However, if you wish to challenge the above account, please provide specific evidence to the contrary, not simply opinions/threats.

As a follow-up to my post from last November about the effort expended trying to get a troublesome PLoS Biology paper dealt with by the journal, I wanted to update on what has happened since (TL,DR nothing!), and address a few peripheral topics along the way…

1) Metrics. At the time of my writing the post, the paper itself had been read 10,000 times and the accompanying PubPeer post 1000 times. So, assuming most people who saw the post on PubPeer clicked through to the article itself, this would suggest  ~10% of the people viewing the paper knew about the problems with it.

A month later (December 10th), the PubPeer page had racked up 2,000 hits and the article itself was at 11,000. In other words, both the paper and the PubPeer page accumulated the same number of hits. Assuming modest overlap, this suggests that in the month after my post, the vast majority of people who saw the paper knew about the problems.

What’s interesting, is although my post generated a modest amount of traffic on Twitter (a dozen or so retweets plus a few responses including from PLoS Biology EiC Theodora Bloom), none of this shows up on the “Discussed” tab at the paper itself. The casual reader would remain blissfully unaware of these problems. I’m sure the journal loves to keep it that way, but it suggests we need better ways of linking social media discussion to publications (note to self – add link to orig’ paper when tweeting about this post).

2) Responses that are not responses.  Within 3 days of my post and the ensuing publicity, PLoS Biology just so happened to publish a blog post entitled “Scientific misconduct allegations: tell me what would you do?“.  It contained some real gems…

I want to be clear that in all the points I make in this post I am not referring to any particular current or past case.

…it is particularly troubling to me that we are seeing a proliferation of websites devoted to anonymous and/or public allegations of misconduct. I am also personally troubled when, as has happened recently, accusers appear to suggest that this journal’s staff are dishonest, lazy, incompetent or otherwise delinquent in our approach to handling these issues, simply because we will not publicly comment on proceedings that, quite rightly, happen in private. My personal request would be that those who consider Twitter or an anonymous blog post the best forum for accusations that may terminate someone’s scientific career instead rethink and try to be patient while investigations take place at an appropriate pace.

Despite the claim that this post was not referring to any particular events, it’s hard to believe it was not motivated in some small way by the specific case I raised just 3 days beforehand.

Naturally, I responded to the post, and PLoS went on the defensive. One specific issue raised was whether they had indeed responded to all my emails. Allow me to clarify… not all of the email correspondence I had with the journal was CC’ed to the EiC Theodora Bloom. Some of it was directly with a senior editor at the journal (who will remain nameless). While the EiC did indeed respond every time (albeit often delayed), the senior editor did not. As outlined in my original post several emails went unanswered.

3) So what now? On December 9th I sent the following email:

It is now 1 month since I wrote a lengthy blog-post about my frustrations with a problematic paper in your journal.  The counter is now at 23 weeks and still there is no word on what’s happening.  When my post was made, the paper had ~10k views and it now has ~11k.  The PubPeer page criticizing it had ~1000 views and now has ~2000.  Another way of looking at those similar traffic levels (1000 views each)…
let’s assume most people who saw the Pub-Peer critique clicked through to the paper itself; that means almost everyone who saw this paper in the past month saw the critique first. Almost no-one is reading this paper “blind” now.

I enjoyed your blog post criticizing certain paths in dealing with problem data, exactly 3 days after I called out these problems on my website and Twitter.  While I realize you may not have been referring to any particular case in your writings, let’s just say it was a curious coincidence.  Referring to one item raised during the ensuing discussion – an apparent disagreement on whether you responded to all emails – checking back through my records, you were not CC’ed on all communications, and there were unanswered e-mails to the Editor (Ines Alvarez-Garcia). Thus, while you personally may have responded, this doesn’t necessarily apply to everyone at the journal.

I think the 6 month anniversary of the publication of this paper might serve as a useful landmark for a follow-up blog post, detailing the lack of action, and the accompanying lack of transparency in this entire process.  It’s OK to take your time, but then it’s also expected you will give concrete reasons for doing so.  I have yet to hear any good reason why this case has not yet been resolved (beyond the classic boilerplate arguments that these things are delicate and take time).  I would hope that by the time I publish a follow up post, there will be some positive results to talk about.

I received the following response from the EiC:

I am currently out of the office at a conference but will ensure we get you a more detailed response later this week.

Followed by this response from the senior editor:

Thank you for your email and apologies for the delayed response. We are actively working on this case and we hope it will be resolved in the near future. However, as we have already mentioned in previous correspondence, we cannot share any additional details with you at this stage.

So much for the extra detail!  That was a month ago and there’s been no word since.  Today is the 6 month anniversary of the publication of the original paper, and we are still no closer to discovering what’s actually going on behind the scenes at PLoS Biology. The take home lesson here is that if you find a problem with published science, you’re better off writing it down on paper and throwing it to sea in a bottle, than you are trying to engage in reasoned constructive conversation with the gatekeepers of information, the journals. It’s 2014 now, and this state of affairs just saddens me.

I’d like to say going out with a bang, but the mix of news round here has been rather middling….

Our joint-PI R01 with Keith Nehrke looks like it’s going to be picked up for funding by NIGMS, which is awesome! However, it seems we have to take a haircut on the budget. We applied for $250k/yr standard modular budget (which as everyone knows just doesn’t buy the amount of science that it used to). We’re going to be funded at $165k/yr (34% cut), split between two labs! Naturally, any money is good, but one can’t help thinking if this is the new normal, even being in the lucky “funded” column doesn’t take away the pressure to write more proposals.

On the topic of proposals, a big area of discussion round these parts and all over tha intarnets is attrition of post-docs (post-docalypse) and the hard time people are having to get a foot on the tenure track ladder.  Here are some “tales from the trenches”…

– After 8 years of post-doc’ my wife’s contract was not renewed. She’s now doing adjunct teaching at a handful of local community colleges, in the hope of landing a TT position some time this millennium. Like several of her colleagues from the same department, she started adjuncting part-time while still post-doc’ing. A neighboring lab’ there just got an A1 grant dinged, so that will possibly add another mouth to the adjuncting trough.

– A former post-doc’ who left last year (PI’s grants ran out) is now teaching at a small college 2 hours’ drive away, and commuting weekly. A neighbor of mine is a post-doc’ split between here and another town 2 hrs. drive away – he has 2 young kids and has to commute every week.

– One of my former grad’ students is now 4 years post-doc’, all along has wanted to get into teaching, and is having an incredibly hard time finding a job despite part-time teaching for over 2 years now.  Plus the major science outreach program here for getting high school students interested in mol bio’ didn’t get renewed, so the post-doc’ who was running that is out looking for “alternative careers”.

– A K-grant applicant in my department got a score but not fundable. His contract was renewed for 6 months to allow him to resubmit. No grant = no job after July. Meanwhile post-doc’ and MD/PhD both got AHA grants trashed (the payline was well into the single digits), and a similar grant from the student also crashed as an NRSA app’.

– 3 very successful former grad’ students back here for interviews this fall, hoping to get recruited as junior faculty after VERY successful post-doc’s (multiple C/N/S papers). Dean says that’s simply not going to happen.

It’s a frickin’ wasteland out there!

Meanwhile, yesterday the house passed the Defense Authorization bill, giving the Department of Defense $544.4 billion in discretionary spending, plus another $80 billion for overseas operations (aka Afghanistan). As this beautiful series of InfoGraphics from Mother Jones shows, 1 in 5 federal tax dollars is defense spending, and the Pentagon’s gas bill is bigger than the entire NIH extramural research budget.

Yes there are problems with our economy beyond just rampant military spending, but it’s so disheartening to see good young scientists struggling for a slice of the pie, and then also see 3 entire pies being pissed down the drain in a place most people couldn’t even locate on a map. Taking a leaf from the defense lobbyists’ book, how do we position science spending as the patriotic thing to do? (rant over).

This post is about the extraordinary lengths one must go to, in order to get a journal or institution to take action on a published paper that contains problematic data. This has been a long time in the making, and is important (to me at least) because it’s both a last ditch attempt to get something done, and the first time I’ve used this website as a forum for such material. The post is in 3 parts: (i) A detailed description of the problems with the paper itself. (ii) A narrative of my attempts to get the scientific record corrected. (iii) Some concluding thoughts on the sorry state of affairs in academic publishing today.

The paper in question is this one… PLoS Biology (2013) 11, e1001603. Effects of Resveratrol and SIRT1 on PGC-1α Activity and Mitochondrial Biogenesis: A Reevaluation. [PMID 23874150]

Part 1 – The Problem

I first came across this paper because it featured prominently in a New York Times article entitled “Exercise in a Pill“. I work in the area of SIRT1 and mitochondrial biology, so I thought surely a paper on these topics featured in NYT must be worth a read. However, upon getting into the paper I rapidly discovered some potential problems with the data – it appeared some of the western blots may have been re-used between different figures, and some other data didn’t follow “best practices” that would be necessary given the bold claims being made. The following images are included to illustrate the some of these anomalies.

In these first 2 examples from Figure 1, it appears as if some of the blot images in panel A and panel B are simply different exposures of the same image, but they’re used to represent different samples, different experimental conditions. The patterns of the bands, their juxtaposition, their shape, various imperfections and spots, all fall into the category of “more similar than would be expected purely by chance”.

The same appears here in Figure 6 (above), in which 2 blots one on top of the other appear to be different exposures of the same image. This one is tricky to pin down, because the 2 proteins of interest (COX IV and cytochrome c) have very similar apparent molecular weights (~15 kDa) on SDS-PAGE, so it’s possible that if the same membrane was used to blot for these 2 proteins, with a strip/re-probe in between, then the same imperfections would come through in the final blot.  The problem is, even if this perfectly good explanation is in-fact the case, it begs the question why would you strip and re-probe for 2 proteins that run at the same weight on a blot? There are some situations in which this is permitted or encouraged (e.g., if you want to measure the phosphorylation status of a protein, you probe with the phospho-X antibody, then strip the blot at re-probe with the total-X antibody, and normalize the former to the latter – see below).  However, when dealing with two separate proteins at almost identical mass, using the same blot twice definitely falls outside of best practices.

In Figure 4, the phospho-versus-total scenario comes up. As described above, the typical way these experiments work is to probe with the phospho-X antibody, then strip the blot and re-probe with the total-X antibody, then normalize phospho-X to total-X. When this is done, we can be sure that the phospho-protein signal actually changed, because the normalization (the total protein) is right there on the same gel. However, in this case the phospho blots have been spliced together, as indicated by the vertical lines in the panels, but the total protein blots have not been spliced.  In-fact, in the top panel, that blot image is 4 separate bands pasted together from 4 separate blots.  Sometimes it happens – you run the samples in the wrong order and you have to rearrange the samples to get them in the “right” order for publishing (IMHO this is lazy, just run the blot again the way you want it to be). If this was the case, wouldn’t the samples be mixed up on all the gels for a particular experiment (assuming they’re all run at the same time)? What we’re asked to believe here, is that the authors ran the phospho-AMPK blot with the samples in one order, and the total-AMPK blot with the samples in another order (the correct one, it seems) and then they only rearranged one set of bands so they all matched up. Needless to say, the potential for “convenient adjustments” to the data during this rearrangement, is not a factor in pure un-spliced blots.

The same then happens in the lower 2 panels for phospho-ACC versus total-ACC (i.e. one is spliced, the other not). But, this one has another problem too… generally speaking phospho protein is a sub-set of total protein. So, you might find that a total-protein antibody will recognize multiple bands (maybe different isoforms of the protein), but then the phospho-specific antibody should be more… well… specific. The odd thing here, is the p-ACC blot has 2 bands, but the total ACC blot (which should include all the phosphorylated and other ACC forms) only has a single band.  This falls in the “just plain weird” category.  There’s also the glaring problem that the shapes/positioning of all the bands between phospho-X and total-X just don’t match up. Clearly these blots originated from different membranes, and while this is not strictly forbidden, it certainly doesn’t count as best practices either.

Finally, there’s this anomaly between Figures 3A and 4C.  It’s hard to be sure, but to me those bands just look too similar, with one possibly being a grossly overexposed version of the other. What’s interesting also, is the lower one is pure black on a pure white background, which makes it impossible to “anchor” the band to its surroundings.  This happens a lot in blots presented in papers – people crank up the contrast and adjust the brightness so their blots appear black/white instead of dark-gray/light-gray.  It might conform to what some people think a blot should look like, but it also introduces the potential for hiding splices which is not there in a grayscale image.  It’s impossible to tell if a band is spliced because when you splice together pure white and pure white there’s no seam.

This does impact the interpretation, because blot densitometry relies on the bands being within the linear part of the dynamic range, wherein black = 100% and white = 0%. As such, any band on a western lot in which the center is completely saturated black is unsuitable for densitometry (it will yield a tableau profile, a clipped peak, instead of a nice Gaussian peak).  Thus, it’s not really possible to believe any quantitative data originating from such blots.

In addition to the above, the paper contains 85 panels of western blot data, and every single one is cropped (“letterboxed”) and presented without any molecular weight markers. A lot of them are over-saturated and unsuitable for densitometry, or contain pure black/white bands which make it impossible to tell if they’ve been spliced. There are also instances in which the same antibody in the same cells recognized 1 band in one panel and 3 bands in a different data panel (why so much variability in what the antibody recognizes?) Altogether, there are a host of problems with the data in this paper, and while they might all have a perfectly good explanation, at the very least some of these things fall into the column of “makes me question the conclusions”. So, what did I do about it?

Part 2 – The Solution (I thought!)

Thankfully, the InterNet has afforded a number of tools enabling readers to comment on the published literature and (one hopes) correct the scientific record. In addition to comment systems at individual journals, sites such as PubPeer and PubMed Commons are pushing the boundaries of Post Publication Peer Review even further, aiming for centralized discussion forums. This is a good thing.  We can all look forward to the day when the integrity of a paper is not judged on the basis of the impact factor of the journal it’s published in, but on the quality and reproducibility of the data inside!

So, having read this paper, I trotted on over to the PLoS Biology website and left a comment, outlining the problems above. Here is a saved (PDF) copy of the comment (July 18th). I even used my real name!  I also Tweeted Gretchen Reynolds, the NYT reporter who wrote the piece about Exercise in a Pill.  She didn’t respond.

You might ask why I can’t just provide a link to the comment itself. Well herein lies the problem – PLoS Biology decided that it violated their terms of service and deleted it. The email conversation went like this:

PLoS: We’d like to remove your comment because we believe it violates our terms.

Me: I don’t think it does. Read the disclaimer I wrote at the end of my comment.

Comment removed

Me: What just happened? Can you explain what you plan to do about this?  Why did you act unilaterally and not allow me to explain why my comment is valid?

Me: Hello, anybody home? Oh, I see, you’re just ignoring me now.

Silence. Nothing beyond the boilerplate that they take comments seriously and will “discuss this with the authors”. I even took pains to remind them of the Streisand Effect, whereupon removing the comment might actually have the opposite of the desired effect. Just for fun, let’a take a look at the PLoS commenting terms and conditions. #7 caught my eye…

Questions about experimental data are appropriate, but need to be phrased in a way that does not imply any misconduct on the part of the authors.

Now go read the comment again, particularly my lengthy disclaimer at the end, in which I specifically state that there’s nothing being accused here, and it’s easy to make mistakes with so many blots in a single paper. The comment did not violate PLoS terms.

This is why I love PubPeer so much. So I trotted over there and uploaded my comment.  It stayed there in all its DOI-linkable goodness, yielding some interesting responses about how the original comment had disappeared from the PLoS website. Note that you can also opt to notify the authors of a paper when you comment on PubPeer – the authors have not responded so far.  I have no idea if PLoS automatically notifies the authors if a comment is left, but if they do then the authors have known about these questions since July 18th.

So what about some metrics? As of today (November 8th), the PubPeer comment has been viewed about 1000 times, and the paper itself about 10,000 times. That means 10% of the people who’ve read this paper at PLoS know about the problems with it. That’s huge! Advertisers would kill for that kind of visibility.  PubPeer has most definitely arrived as a format.

I emailed PLoS Biology (the editor who contacted me about the comment) on August 29th (6 weeks gone), asking for a progress report. This time I CC’ed the editor-in-chief. They responded with a boilerplate about how they comply with COPE guidelines.

I emailed PLos Biology again on September 26th (10 weeks). No response.

I emailed PLoS Biology again on October 3rd, asking for a status update. In this email I linked to the PubPeer comment. I also mentioned that Derek Lowe at the amazing “In the Pipeline” blog had written about the paper and its apparent problems. His site gets around 15-20 thousand hits per day. No response from PLoS.

I emailed PLoS Biology again on October 16th with the following…

Nearly 13 weeks (over 3 months) and counting.
I wonder how much longer we will be expected to wait for a resolution to this quite simple case? The courtesy of a response is requested.
PSB

No response. In addition to taking a ridiculously long time to deal with this problem (which thousands of people know about), now they’re just being plain rude and refusing to answer my emails.

So what else can be done?  Well thankfully this October saw the launch of PubMed Commons, a shiny new commenting system that’s part of NCBI’s PubMed. Aha! maybe this will work where all else has failed?  I happily trotted along and uploaded my comments there. Unfortunately comments are only visible to those logged in via @myNCBI, so if you go to the PubMed page for the paper and log in from there, you should be able to see the comment. Again, I don’t know the inner workings of PubMed Commons, but I would imagine it has to contain some sort of notification system to the journal or the authors. It’s therefore interesting that neither PLoS Biology or the authors have been in touch regarding my comment (using my real name) on PubMed Commons.

As of today (November 8th), it has been 16 weeks since I first raised problems about this paper, and it’s still out there unscathed in the literature, with no indication of any problem whatsoever on PubMed or PLoS Biology’s websites. If I was the pessimistic type, I could take those numbers further up the page and say that 90% of the people reading this paper have no idea about the data problems in it. That’s sad, not to mention dangerous.

Part 3 – What Does it Take?

So here’s the question – what in the name of fuckingcockbuggeringdogshit does it take to get a problem paper dealt with in this day and age?

• I commented on the journal’s own website
• I left comments on PubPeer
• I tweeted about it
• I left comments on PubMed Commons
• It’s been featured on a 15-20k hits/day blog
• I emailed the journal editors until I was blue in the face
• 10% of the people who’ve seen the paper know about the problems

I’m out of options. If a journal can take this much pressure and just brush it all off, what’s the point of all these post publication peer review systems? It should not take this much effort on the part of a reader, to correct the scientific record!  It should not take a scientist having to use his own personal lab’ website as the last resort to communicate his frustrations about a totally failed system.

Does anyone agree with me that 16 weeks (nearly 4 months) is too long to establish the authenticity of a few dodgy looking blots?

My lab does a lot of western blotting, and if I could not find you an original film within 24 hours I would be ashamed. I don’t think anyone who calls themselves a real scientist would have trouble finding original data within a couple of days at most.  It is completely beyond my understanding why this is taking so long.

Is PLoS Biology really so short-staffed they can’t devote someone to analyze these data for a few hours? PLoS is a $23m/yr. operation; hire more ethics staff already!  Do they not have a policy which says “anyone who can’t provide us with original data on demand will have their work retracted”?  Don’t they have the ability to put an expression of concern on the paper while it is being investigated? Are they really so rude they can just stop responding to email? Are they really so naive as to think they can ignore the combined power of email, PubPeer, PubMed Commons, Twitter, the blogosphere and at least 10% of an article’s readership? Are they delusional enough to think this problem will disappear if they put on a poker face?

How did we arrive at this totally screwed up system in which the gatekeeper journals decide what the truth is, and their scientist paying customers have to expend enormous personal effort just to get a dodgy looking piece of data explained properly? There has to be a better way.

Finally, I hope it doesn’t need to be spelled out, but the point here is to make a critique on the sorry state of academic publishing, and not to focus on the origins of the problems in the paper itself. There is of course the possibility that I might be wrong. The authors could be sitting on a treasure trove of original data, and they’re going to produce it all for the editors at PLoS Biology to see and this will explain all the anomalies. Maybe a correction will ensue, or maybe PLoS will deem this unnecessary and just close the case. There’s probably no way I’ll ever get to see the original data (that’s not the way these investigations work), so I may just have to take the journal’s word if they say “move along nothing to see”. That’s OK. If I’m wrong on this I promise to apologize profusely for any trouble I’ve caused the authors. Hey, I might even send them a peace offering (authors – if you’re reading this, let me know your beverage of choice).

The bigger point here, is that regardless the outcome of this particular case, it should not take this much time / effort / energy / frustration. The problem is not dodgy looking data, it’s the entire system we have for dealing with it. So don’t focus on this one paper. Instead, recognize that it’s only one example of hundreds that myself and a few like-minded people deal with on a regular basis. It’s a representative case, meant to illustrate a deeper problem.

Thanks for reading.