Today marked a new low in the depths to which publishing industry will sink to maintain their outdated business model.  I’m talking about a piece of literary detritus entitled “Vigilante Science” published in Plant Physiology and penned by its editor in chief Michael Blatt. It’s a classic punch-down piece from a journal editor having a hard time coming to terms with the death of the publishing industry as we know it.

Readers of this op-ed masquerading as serious discourse will rapidly learn that the popular post-publication peer review portal PubPeer (8 Ps – count em!) is a mere “by-product of the social media age”. Straight away, he starts out dissing it as side-issue, as opposed to what it really is – a potential solution to a rampant problem foisted on academia by multi-billion dollar publishing corporations (one of which he happens to work for).

Readers are then treated to (wait for it…) the two-fold problems with PubPeer…

Problem 1 is simply stated as “Most commenters take advantage of the anonymity afforded by the site in full knowledge that their posts will be available to the public at large”.  There is no further attempt to explain why this is a problem, we’re just asked to assume that it is.

Problem 2 is stated bluntly as “The vast majority of comments that are posted focus on image data (gels, blots, and micrographs) that contribute to the development of scientific ideas but are not ideas in themselves”.

After stepping back from my monitor to avoid spitting coffee at it, I read that sentence again, and have read it several times since, and I can’t even begin to understand how someone purporting (pretending) to be a scientist could think in such a way?

To borrow another of Dr. Blatt’s phrases – “Let’s not mince words”.  What we’re talking about here is the editor of a major journal, claiming that problems with data are not a big deal, as long as the underlying ideas are OK.  Hopefully I don’t need to explain how dangerously at odds with the scientific method such a viewpoint is (hint – it’s all about the data; anyone can have ideas, but without data to support them, ideas are worthless).

Taken together, the two-fold problems (anonymity and mostly petty issues with figures) pose fundamental paradox:  If PubPeer is really as unimportant as he claims it is, why bother with an editorial such as this?

The good Dr. Blatt’s justification for his tirade against commenters who have an eye for dodgy data, is that “no journal club I ever organized or contributed to was so obsessed with the minutiae of data presentation”.  My only response can be to suggest he find a better journal club!

The article moves on with a series of poorly informed opinions. Take for example this gem…

“While there is no danger of public embarrassment for the commenter, likewise there is no opportunity to gain from a personal exchange with the author.

Does Dr. Blatt even consider the possibility that many commenters on PubPeer have already exhausted the polite channels he espouses?  Does he consider the fact that when confronted with problem data, many authors simply ignore, obfuscate, or go on the offensive (often hiring lawyers)?  The opportunity to interact with authors is presented as a panacea, failing to acknowledge the reality that such interaction is often fruitless.

He continues…

What is the rationale? Given that the majority of comments show the most petty kind of scientific criticism, can there be any doubt that the intent often is to pillory”.

Does it even cross his mind that a possible motivator might be the integrity of the scientific record? Does he stop to think that commenters may be tired of trying to compete with cheats?  No, it’s far easier to write it all off as a bunch of disgruntled scientific competitors whose sole motivation is the satisfaction of tanking someone’s career!  To put it mildly – many of us have bigger fish to fry (i.e. our own science).

This little gem seeks to put pre-publication peer review (i.e. the bread-and-butter of his journal) on a podium, somehow better than post-pub…

“I accept that there is a case for anonymity as part of the peer-review process. However, the argument for anonymity in postpublication discussion fallaciously equates such discussion with prepublication peer review.”

Given that many of the cases on PubPeer have found problems that should have been discovered during pre-pub’ peer review, there is a strong argument to be made that post- review is not only equal to pre-, but better!  Considering that both exercises are conducted for free, but the only one making a profit on the back of free labor is the pre-pub’ variety, perhaps we should equate them by paying people for post-pub’ peer review?

It goes on… he sides with Hilda Bastian in espousing “the importance of assessing whether commenters are outside their areas of expertise”.  This is a classic prat-fall of the entitled.  I’ll re-phrase it into plain English – Your opinion only counts if I deem you important enough to have an opinion.  Witness this discussion between myself and a senior scientist on PubPeer, in which my scientific credentials were considered as a topic worthy of discussion, instead of the actual data in question.  Quite simply, there are no rules regarding who is qualified to comment on science. Anyone who claims otherwise is engaged in protectionism/racketeering.

He then adds this beauty…

“So, whatever the shortfalls of the peer-review process, I do not accept the argument that it is failing, that it is a threat to progress, or that, as scientists, we need to retake control of our profession. Indeed, if there is a threat to the scientific process, I would argue that, unchecked, the most serious is the brand of vigilante science currently facilitated by PubPeer.”

So let’s get this straight – the problems facing science today are not: (i) a lack of funding,  (ii) rampant fakery, (iii) politicians seeking to defund things they don’t like, (iv) inadequate teaching of the scientific method in schools, (v) proliferation of the blood-sucking profiteering publishing industry, (vi) an obsession with impact factor and other outdated metrics, (vii) a broken training to job pipeline in academia, (viii) insert your favorite #scipocalypse cause here.

No, the #1 threat to science right now is vigilantes on PubPeer! Of course… if only we could just get rid of the idiots on PubPeer, the most serious threat to science would disappear and we could all go back to living the high life!  Sign me up!

He then issues a dangerous call to action…

“I urge scientists publishing in Plant Physiology and other reputable scientific journals not to respond to comments or allegations on PubPeer”.

My only suggestion here would be to ignore this “advice”, unless of course you plan on a long and protracted series of legal challenges or online battles, resulting in an institutional investigation and your eventual firing.

The way that “real” scientists respond when their data is questioned, is to answer the damn question!  Show the data. Produce the originals.  In case you hadn’t noticed, the front page of PubPeer cycles once every 3-4 days – if there’s an innocent explanation, you WILL be vindicated and your career will not end if you engage with the commenters.

To cap it off, in the ultimate punch-down, Dr. Blatt accuses the founders of PubPeer of unmasking themselves solely for the purpose of making money. This neatly sidesteps the huge amount of personal resources they poured into the enterprise from the beginning, and of course ignores that they will not gain anything personally from this new effort, because the organization is a 501c3 non-profit foundation.

It takes an exquisite amount of hipocrisy, to speak from the bully pulpit of an entitled publication, part of a multi-billion dollar enterprise, punching down at a non-profit foundation, and accusing it of being money-hungry.  The only possible motivation I can think of for this Op-Ed, is an editor and an industry witnessing the slow decentralization of their control over information (for massive profit), seeking to discredit an upstart grass-roots organization that might disrupt the status quo.

What a sad sad display of the death throes of an empire.

Such events usually do not end in the emperor’s favor.

Finally, our metabolomics paper is in press at J. Mol. Cell. Cardiol.  (email if you want a reprint).  TL/DR… SIRT1 drives most (~85%) of the metabolic alterations that occur in the heart during acute ischemic preconditioning (IPC).

This was quite a tough paper to get published. We started the project in spring 2013, and wrote it up in fall 2014. It got rejected from a big journal (IF>15) first, then went 2 rounds at a mid-level (IF>10) journal before being rejected again, and then it went 2 rounds at JMCC before acceptance. All told, a year of back and forth with reviewers and editors.

The model system we used to investigate this topic was the Langendorff perfused mouse heart and splitomicin, a pharmacologic inhibitor of SIRT1. The basic issue with the reviews that ended up as rejections, was an insistence by reviewers that we do things in-vivo and using knockout mice.

Normally, we’re big fans of moving toward more physiologically-relevant model systems, but in this case there are very specific reasons to use a perfused heart and a pharmacologic inhibitor.  Here are some key points…

(1) Regarding pharmacology, the inhibitor we used is one we’d already shown can block acute IPC, so it’s a good candidate to test whether it also blocks the metabolic effects of IPC. Also, we had already shown that a SIRT1 KO mouse heart cannot be preconditioned, and that the endogenous protection seen in the SIRT1 over-expressing transgenic mouse can be blocked by 5 min. infusion of the inhibitor. Thus, the time-frame for the effects of SIRT1 in IPC is very short – on the order of 20 min. The SIRT1 KO mouse has known long-term metabolic alterations which would mask any changes we’d look for in IPC.

(2) Regarding in-vivo vs. in-vitro, it all boils down to sampling time. In our system, we can clamp the heart in liquid nitrogen Wollenberger tongs, straight off the perfusion rig. In effect, it goes from beating to frozen in less than a second. That’s important for getting reliable information on metabolites such as ATP, NADH, GSH and other labile redox things.

The problem is, when you precondition a mouse heart in-vivo, it’s a focal ischemia model. Only part of the heart is ischemic (the bit downstream of the vessel you occlude), so if you try to dissect out the ischemic zone, you delay the clamping by a couple of minutes and destroy all the labile metabolites during the dissection. Alternatively, if you clamp the whole heart right out of the animal into liquid nitrogen you create 2 problems… First, all the changes in the ischemic area get “diluted” with the other part of the heart that wasn’t ischemic (the so-called “area not at risk”).  Second, you’re also sampling blood, so you don’t know if the changes you see are in the myocardial tissue or the blood that comes along for the ride (by our estimates when you clamp a heart out of a mouse, about 1/3 of the sample is blood). In contrast, the perfused heart system has no blood, so the whole sample is myocardium. Also the entire heart is ischemic, so there’s no dilution.

(3) The other major issue concerns the type of metabolomics analysis you want to perform.  In this paper, we performed not only steady-state metabolomics (i.e., measuring the relative levels of metabolites), but also ¹³C labeled substrate tracing. The latter can yield proxy information about metabolic flux, which steady-state measurements cannot. This is easy in the perfused system… just throw ¹³C-glucose or ¹³C-palmitate in the perfusion media, but in-vivo this creates problems. You can’t just deliver labeled substrate to a whole mouse and assume it’s only being metabolized by the heart on first pass.  For example, the cardiac/liver Randle cycle can result in labeled glucose being turned to labeled fat by the liver, then sent to the heart as fuel. Also, whatever ¹³C-substrate you infuse is going to compete with endogenous blood-borne substrates in the animal. In the perfused system you can swap out the whole substrate (i.e., replace all the glucose with ¹³C-glucose), so you have much tighter control over delivery.

So, this really is one of those cases where an abstract application of Krogh’s principle comes into play.  The in-vitro and pharmacology based approach really was the best system available to answer the question at hand (that question being, what fraction of the metabolic changes that occur in acute IPC are governed by SIRT1 signaling?)

Naturally, we argued all the above points and it didn’t get us anywhere! As a lab that routinely uses both in-vivo and knockout models, it’s rather frustrating to be locked out of publishing in certain journals because we chose to use an allegedly inferior system. It’s annoying that some journals have a myopic focus on knockouts and in-vivo data which precludes them from publishing otherwise solid work.  Thankfully JMCC seems to have a more sensible approach to this type of work!

Last week I had the honor of being a speaker at the “MiP” (mitochondrial physiology) school, in Greenville NC. The event is one of a long series organized by Erich Gnaiger (inventor of the Oroboros Oxygraph2k respirometry apparatus).  The meeting included a series of methods workshops and scientific talks from abstracts, as well as didactic lectures based on a framework of the book “Bioenergetics 4“, one of whose authors (David Nicholls) gave several lectures.

My lecture was on “Mitochondrial ROS generation”, a seemingly massive topic which cannot really be covered in any depth in 45 minutes.  But anyway, here are the slides (PDF), in case anyone might find them useful.

It’s been a slow summer so far in the lab, but we’ve had a few personnel changes and other things to update…

– Marcin Karcz finished his 1 year research residency funded by FAER, and has now moved on to a cardiac anesthesia fellowship at Columbia NY.

– We have a new lab’ tech, James Miller, who joins us from the Department for Oral Biology.

– Our new post-doc’ fellow Yves Wang started July 1.  Yves joins us from Case Western in Cleveland, where he’s worked on cardiac imaging until now.

– We’re having “fun” trying to get stuff published.  Currently we have 3 papers out there going through the wringer – one on anesthetic preconditioning, one on SIRT1 and cardiac metabolism, and another on autophagy.

– We filed a provisional patent on one of our drugs that protects the heart from IR injury when delivered at the moment of reperfusion, at a concentration of only 10 nM!  The paper on this is being written up for submission in the fall.

– Too many awesome meetings this year…. Jimmy & Sergiy at the AHA BCVS meeting in New Orleans this month. Paul & Owen are going to the TRiMAD meeting in State College PA this November. Paul is also presenting at the APS Physiological Bioenergetics meeting in Tampa in September, and the MiP bioenergetics workshop in Greenville NC in August, and attending the Society for Heart & Vascular Metabolism (SHVM) meeting in NY in October. That’s not counting the AHA sessions and SFRBM Annual meeting, both in November. This has been a seriously big year for conferences!

Recently, Cell editor in chief Emilie Marcus posted an article from the perspective of an editor, decrying the recent uptick in allegations of data mis-handling that appear to be flooding editors’ desks in recent years, and asking what to do about it. I’ve already opined on how Cell royally screwed up this process in the past, so instead let’s focus on the actual questions asked, and some solutions. Here are the specific questions posed by Dr. Marcus…

(1) At a time when there is increasing pressure to reduce the costs of publishing, how should journals, institutions, and funders apportion time and resources to addressing a burgeoning number of alerts to potential instances of misrepresentation or misconduct?

Charge less and spend more?  Scientific publishing is a multi billion dollar industry, with profit margins routinely in the region of 35%. All the labor (i.e., peer review) is essentially free, and the actual production costs are miminal now that everything is on-line.

The issue here is that Cell and the other publishers want it both ways – they like the title “gatekeepers of truth”, but they don’t want to shell out the cash required to ensure that what they’re peddling is actually true!  Seriously, the answer to this one is so simple, it really questions the sanity of anyone asking it – SPEND MORE MONEY!  Hire more people to scrutinize the data before it goes out the door.  In the life sciences field in the US right now, we’re all bemoaning the glut of former grad students and post-doc’s struggling to find jobs. They have intimate knowledge of the subject matter. Hire them and you’d fix 2 problems in science.

(2) Are there ways to improve the signal-to-noise ratio that we haven’t thought of?

By signal to noise, one presumes Dr. Marcus is referring to the number of allegations that come in the door, but then don’t turn out to be real problem data.  Again this is a tractable problem.  More eyes on the data = easier to figure out what’s real.  Hire more trained eyes.

Another really simple solution is to USE THE TOOLS ALREADY AVAILABLE for plagiearism detection – things such as iThenticate and DejaVu.  It is shocking that we’ve had text plagiarism software now for well over a decade, but most journals simply don’t use it. Why? Because it costs money!  What’s interesting is that software tools are now also being developed to do the same thing for data and images (I know of some in the pipeline, but can’t mention specifics).

So, we are literally a couple of years away from the point when any submitted paper – both text and images – can be screened automatically using software. After that watershed moment, does anyone want to gamble how long it will take before such tools gain widespread adoption across the publishing industry?  Don’t hold your breath!

The other answer to this question is to think differently about the problem of signal:noise. In the old days, the only way to do it was boost the signal and cut the noise.  Computing power and labor are now so cheap, it’s easier to just do everything – take the signal, take the noise, take everything, and look at it all.  If there’s some noise in there, who cares? It’s actually more expensive to expend effort trying to figure out what’s noise. Just examine everything and THEN decide if it’s noise. You don’t gain anything by filtering out the noise first.

(3) Is there a process for investigating that would be more streamlined, coordinated and efficient?

Yes, see above. Software and more pairs of eyes. Spend more money (does anyone see a theme developing here?)  As regards efficiency, that fact that it was necessary to ask all of these questions tells you that the current system is not working. Quite frankly, anything the publishers do differently will be more efficient than the current approach.

(4) Would allowing/requiring authors to post all the raw data at the time of publication help?

Yes, both of these. It simply beggars belief that in 2015, when I can fit the entire back catalog of thousands of journals on a pocket-sized hard disk that costs <$100, journals are still on the fence about whether to allow authors to store all the data associated with a paper.  Hell, they’re still imposing limits on word count, pages, number of images etc.

Data is, quite literally, the cheapest possible thing in the world that you can store. Journals need to get out of the last century and embrace scientists’ wish to both include all their own data, and to see all the data in other people’s papers.

(5) Should we require that all whistleblowers be non-anonymous to ensure accountability? What if we enact this policy and need to let a serious claim go unaddressed because the whistleblower refuses to reveal their identity?

“Facts should be viewed as such, regardless where they came from”.  As a society, we are in grave danger when we attach relative importance to facts dependent on the perceived importance of the messenger. I experienced this first hand on PubPeer, when a scientist whose work I questioned went on a long diatribe about my own qualifications – as if that somehow changed the facts of the case.

Although the term used in Dr. Marcus question here was “accountability”, inherent is the assumption that anonymity equates to unreliability. There are anecdotes about the infamous Clare Francis being wrong much of the time, but that’s N=1 and I think we can do slightly better here. PubPeer has admirably demonstrated on thousands of occasions that anonymous reporters should be taken seriously, because they are very often right. Similarly, informants who use their real names are very often wrong. There is no hard evidence (to the best of my knowledge) that the reliability of allegations is correlated with the named status of the accuser.

The danger with a “named” approach is the slippery slope to an importance of the messenger prat-fall. Will a journal take an allegation more or less seriously if it comes from a post-doc’ versus a senior PI?  What about an undergrad?  What about a non-scientist member of the public? What if the accuser is a former employee of the paper’s author – does that somehow disqualify their opinion, or does it make their accusations more valid because they may have first hand knowledge of the case?  All these examples lead to a simple conclusion – identity does not matter.

If a journal chooses to assign “importance” to a series of allegations based on who they came from, one must assume that a similar biased system will exist at the other end of the investigation, i.e., the journal may choose to take allegations seriously or not depending on the status of the scientist who is being accused.  If a journal had a policy stating “we don’t investigate Nobel prize winners”, that would be offensive. Why is ignoring anonymous reporters any less offensive?  Both strategies attach undue importance to the messenger, not the facts.

(6) Should we only consider concerns related to papers published in the last 5 years? Seems fine for “small things” like gel splicing, etc., but presumably, if a concern arose that some body of work was fraudulent, even if it was 10 years old, wouldn’t we want to correct the published record?

A possible strategy might be to investigate everything fully within a given time frame (say 6 years – that’s the ORI statute of limitations), but then for older papers apply a graded approach dependent on the other papers from an author or group.

For example, if a paper from 10 years ago is questioned, and the problem is un-reported gel splicing, this may indeed represent a simple mistake by the authors who were simply following accepted (and now universally acknowledged to be wrong) contemporary practices.

However, that same paper juxtaposed against a back-drop of 20 similar papers, all with problems, perhaps with a few already retracted or corrected, suggests a pattern that may be indicative of misconduct or (at the very least), sloppy data handling habits.

___________

This last point is where another key solution comes in… COORDINATION.  None of the above proposals will work if each journal tries to implement them individually.  There has to be a database, shared between journals and publishers, to keep track of all these problems.  The simple idea would be as soon as an allegation comes in, the journal’s investigator would look up the authors in the database and see if any other journals had active on-going investigations about the same authors. Right now, doing a search on PubPeer, PubMedCommons, and Retraction Watch is a reasonable proxy for this, but far from comprehensive or perfect.

Finally, another potential solution that has been overlooked is the role that funding agencies have to play in this process. In case anyone didn’t notice, the NIH open access mandate was introduced in 2008, and sat pretty much ignored for a couple of years. Then NIH came up with a simple carrot/stick – if you don’t comply, you won’t get funding.  Boom!  Now everyone is super careful to publish their work in journals that comply.

What if NIH were to draft up a set of regulations for how journals ought to deal with these problems?  Not guidelines or recommendations but actual RULES… Want to publish your NIH-funded work? It has to be in a journal that plays by the rules, otherwise you can’t list it on your bio-sketch.  Just watch how quickly all the journals would fall into line. Their bread-and-butter is packaging up science they didn’t pay for and selling it back to the public, so if the hand that feeds them the bread says “jump”, they will jump. A mass exodus of researchers from journals that don’t comply, would be cool.  Maybe NIH could call the new mandate “Regulations On Biomedical Oversight Concerning Obsolete Publishing Practices”