A Skeptic Over Coffee: Young Blood Part Duh

Does this cloudy liquid hold the secret to vitality in your first 100 years and beyond? I can’t say for sure that it doesn’t. What I can say is that I would happily sell it to you for $8,000.

Next time someone tries to charge you a premium to intravenously imbibe someone else’s blood plasma, you have my permission to tell them no thanks. Unless there’s a chance that it is fake, then it might be worth doing.

Californian company Ambrosia LLC has been making the rounds in publications like the New Scientist hype-machine to promote claims that their plasma transfusions show efficacy at treating symptomatic biomarkers of aging. Set up primarily to exploit rich people by exploiting younger, poorer people on the off chance that the Precious Bodily Fluids of the latter will invigorate the former, the small biotech firm performed a tiny study of over-35s receiving blood plasma transfusions from younger people. It’s listed on clinicaltrials.gov and everything.

First of all, to determine the efficacy of a treatment it’s important that both the doctors and the patients are blinded to whether they are administering/being administered the active therapeutic. That goes all the way up the line from the responsible physician to the phlebotomist to the statistician analyzing the data. But to blind patients and researchers the study must include a control group receiving a placebo treatment, which in this case there was not. So it’s got that going for it.

To be fair, this isn’t actually bad science. For that to be true, it would have to be actual science. Not only does a study like this require a control to account for any placebo effect*, but the changes reported for the various biomarkers may be well within common fluctuations.

Finally, remember that if you assess 20 biomarkers with the common confidence cutoff of p=0.05, chances are one of the twenty will show a statistical difference from baseline. That is the definition of a p-value at that level: a 1 in 20 chance of a difference being down to random chance. Quartz reports the Ambrosia study looked at about 100 different biomarkers and mentions positive changes in 3 of them. I don’t know if they performed statistical tests at a cutoff level of 0.05, but if so you should expect on average 5 of 100 biomarkers in a screen to show a statistical difference. This isn’t the first case of questionable statistics selling fountain of youth concepts.

All of this is not to say that the experiments disprove the positive effects of shooting up teenage PBFs. It also generated zero conclusive evidence against the presence of a large population of English teapots in erratic orbits around Saturn.

You could conclude by saying “more scientific investigation is warranted” but that would imply the work so far was science.

* The placebo effect can even apply to as seemingly objective a treatment as surgery. Take this 2013 study that found no statistical difference in the outcomes of patients with knee problems treated with either arthroscopic surgery or a surgeon pretending to perform the surgery.

I

Is the future of scientific publishing in-house open access?

WeirdFuture
Photo from flickr user Tom Marxchivist, 1952 cover by Basil Wolverton, used under CC attribution license.

Those of you that frequent theScinder know that I am pretty passionate about how science is disseminated, and you have probably noticed that, like their brethren in newsprint and magazine before them, the big-name publishers don’t know exactly how to react to a changing future, and despite what traditional publishers would have you believe, they are not immune to publishing tripe.

Nature may be butting heads with Duke University over requesting waivers for the open access policy in place there. Apparently the waiver request isn’t even necessarily based on the practical implementation of Duke’s open access policy (Nature allows articles to be made freely available in their final version 6 months after publication), but it does raise the question: how much hassle will universities and their faculty put up with before they take matters into their own hands? As MIT’s Samuel Gershman points out, modern publishing doesn’t cost all that much. Even the fairly exorbitant fees charged to authors by the “gold standard” of open access publishers may be a transient relic of the conventional (turning archaic?) publishing business model. This provides incentive for predatory publishing (as discussed in this article at The Scientist and the basis for the Bohannon article published in Science last October) But if peer review and editing is largely volunteer labour, performed as an essential component of the role of a researcher and with the bill largely footed as a public expenditure, why keep paying enormous subscription fees for traditional publishing? If the trend catches on, as it almost certainly will, leading institutions will continue to adopt open access policies and libraries will see less and less reason to keep paying for outdated subscriptions.

Relevant links:

Scholarly Publishing: Where is Plan B?

California univerisity system consider boycotting Nature Publishing Group

Samuel Gershman’s ideal publishing model, the Journal of Machine Learning Research

Open Access Death Knell. Or Is It?

Is_this_tomorrowrev
I told you publication was a fiat currency

Last week, Science published a critical investigation into the state of peer review in open access journals. John Bohannon, the author,  generated a faux paper describing a set of fictional experiments testing the effects of secondary metabolites from lichen on cultured cancer cells. These papers were sent to a selection of open access journals sampled from the Directory of Open Access Journals (DOAJ) and Larry Beall’s infamous list. The lichen species, secondary metabolite, and cancer cell line were varied randomly to generate iterations of the paper with slight differences, but the shoddy results were the same in each. In a few of the iterations I examined, the structures didn’t even match the secondary metabolite described. The researchers didn’t exist, their names random combinations from a database of first and last names with random middle initials, and the institutions they worked at were fictional. A “dose-dependent” growth inhibition effect in ethanol buffer (with no EtOH in controls) spans five orders of magnitude, and shows growth rates all with overlapping confidence intervals at 95%.

Of 304 papers submitted to various open access journals, 157 were accepted, many of them without any real review taking place. 98 were rejected, and 49 were still up in the air at press time. The article seems to make a solid case against the relatively nascent open access model of publishing, and that is certainly the tone represented by the article and associated press coverage. However, if I assume that the average reader of Science is scientifically literate, then I would expect that most readers will remain unconvinced that open access is dead and dangerous.

In Who’s Afraid of Peer Review Bohannon combines language from both scientific and journalistic writing styles, taking advantage of the credibility implied by describing sample-selection and procedural decisions in a semi-scientific manner, as well as the journalist’s ability to make general claims with a strength that would be questionable in a scientific article.

Examples:

And the acceptances and rejections of the paper provide the first global snapshot of peer review across the open-access scientific enterprise.

137 of the journals chosen for this investigation were pulled from a black list maintained by Jeffrey Beall at the University of Colorado Boulder. In places (such as the general description of results) the overlap between Beall’s list and the journals selected from the DOAJ is not clear. In the original sample, 16 of these journals are in both the DOAJ and Beall’s list, but it is difficult to tell if they made it into the final analysis because 49 of the 304 journals selected for submission were thrown out for “appearing derelict” or failing to complete the article review by press time.

For the publishers on his [Beall’s] list that completed the review process, 82% accepted the paper. Of course that also means that almost one in five on his list did the right thing—at least with my submission. A bigger surprise is that for DOAJ publishers that completed the review process, 45% accepted the bogus paper.

This is somewhat misleading, as it implies that the 45% and 82% results are exclusive of each other. I could not tell just from reading the paper what proportion of the 16 journals found in both Beall’s list and the DOAJ made it to the final analysis. Furthermore, I know this is misleading based on how Jeffrey Beall, who is quite close to the subject, interpreted it: “Unfortunately, for journals on DOAJ but not on my list, the study found that 45% of them accepted the bogus paper, a poor indicator for scholarly open-access publishing overall.”

Acceptance was the norm, not the exception.

157/304 journals (51.64%) accepted the paper. While this is a majority, I would hardly qualify acceptance as a typical result when the split is so nearly even, especially when 137 of the 304 journals had already been blacklisted. Discrediting open access in general based on the results reported is not a fair conclusion.

Overall, the article just misses making a strong critical statement about the state of scientific publication, instead focusing only on problems with predatory publishing in open access. By ignoring traditional journals, we are left without a comparison to inform what may be quite necessary reform in scientific publishing. Bohannon’s article is likely to be seen and read by a large number of people in both science and scientific publishing. Editors can be expected to be on alert for the sort of fake paper used by Bohannon and Science, making any comparison to traditional publishing models just about impossible for now. Finally, the overall effect is to damn innovation in publishing, particularly open access models, and it is not surprising that the sting article was published by the “king-of-the-hill” of traditional scientific journals. It is possible that the backlash against open access and publishing innovation in general will actually impede necessary progress in scientific publishing.

As long as an academic career is judged blindly on marketing metrics such as publication frequency and researchers continue to accept excessive publication fees, there will remain an incentive for grey market “paper-mills” to gather up unpublishable papers for profit. Overall, the open access model has thus far incorporated too much from traditional publishing and not enough from the open source movement.

Science magazine warns you that open access is too open, I say that open access is not too open enough.

text in block quotes is from Who’s Afraid of Peer Review by John Bohannon, Science, Oct. 4 2013

Original version of image here

EDIT: link to John Bohannon’s article