­A Skeptic Over Coffee – Young Blood

dsc_0005

A tragic tale of a star-crossed pair,
science vs. a journalist’s flare

When reporting on scientific topics, particularly when describing individual papers, how important is it for the popular coverage to have anything to do with the source material? Let’s take a look at a recent science paper from Justin Rebo and others in Nature Communications and the accompanying coverage by Claire Maldarelli at Popular Science

Interest in parabiosis has increased recently due to coverage of scientific papers describing promising results in mice and the high-profile of some parabiosis enthusiasts. Parabiosis, from the Latin for “living beside”, typically has involved stitching two mice together. After a few days the fused tissue provides blood exchange through a network of newly formed capillaries.

The most recent investigation into the healing effects of youthful blood exchange from Rebo et al. expands the equipment list used for blood exchange beyond the old technique of duct-taping two animals together surgically joining two animals. Instead of relying on the animals to grow new capillary beds for blood exchange to occur, the authors of the new paper used a small pump to exchange a few drops of blood at a time until both mice had approximately the same proportion of their own blood and that of a donor and vice-versa.

According to the coverage from Popular Science:

While infusing blood from a younger mouse into an older mouse had no effect on the elderly mouse in the latest study, infusing blood from an older mouse into a younger one caused a host of problems in organs and other tissues.

Just a few paragraphs further Maldarelli quotes Conboy (last author on the paper) as saying “‘This study tells us that young blood, by itself, cannot work as medicine’.” In contrast, in the paper the authors state that “Importantly, our work on rodent blood exchange establishes that blood age has virtually immediate effects on regeneration of all three germ layer derivatives.” and later that “. . . extracorporeal blood manipulation provides a modality of rapid translation to human clinical intervention.”[1] There seems to be a bit of disagreement between the version of Conboy on the author list of the scientific article and the version of Conboy quoted in the PopSci coverage of the same article.

We also learned from Maldarelli that the tests reported in the paper were performed a month after completing the blood exchange procedure, but the longest duration from blood exchange to the experiment’s end (sacrifice for post-mortem tissue analysis) was 6 days after blood exchange.

I came across the PopSci coverage when it appeared on a meta-news site that highlights popular web articles, so it’s safe to assume I wasn’t the first to read it. Shouldn’t the coverage of scientific articles reported in the lay press have more in common with the source material than just buzzwords? The science wasn’t strictly cut and dried: not every marker or metric responded in the same way to the old/young blood exchange, and while I agree that we shouldn’t be encouraging anyone to build a blood-exchange rejuvenation pod in their garage, the findings of the article fell a long way from the conclusions reported in the lay-article: that young blood had no effect on the physiology of old mice. This is to say nothing about the quality of the paper itself and the confidence we should assign to the experimental results in the first place: with 12 mice total* and a p-value cutoff of 0.05 (1 out of every 20 experiments will appear significant at random), I’d take the original results with a grain of salt as well.

This is the face of science we show the public, and it’s unreliable. It is no easy task for journalists to accurately report and interpret scientific research. Deadlines are tight, and writers face competition and pressure from cheap amateur blogs and regurgitation feeds. “What can I do to help?” you ask. As a consumer of information you can demand scientific literacy in the science news you consume. Ask for writers to convey confidence and probability in a consisent way that can be understood and compared to other results by non-specialists. As a bare minimum, science and the press that covers it should at least have more in common than the latest brand of esoteric jargon.

If we only pay attention to the most outlandish scientific results, then most scientific results will be outlandish.

*The methods describe a purchase of 6 old and 6 young mice. However, elsewhere in the paper the groups are said to contain 8 mice each. Thus it is not clear how many mice in total were used in these experiments, and how they managed to create 12 blood exchange pairings for both control and experimental groups without re-using the same mice.

[1] Rebo, J. et al. A single heterochronic blood exchange reveals rapid inhibition of multiple tissues by old blood. Nat. Commun. 7, 13363 doi: 10.1038/ncomms13363 (2016).

We mustn’t liken a black hole to a baked good

We also mustn’t use the royal we

A prevalent mindset in science journalism is that in order to make a subject accessible, it first must be dumbed-down. I suggest we all make efforts to recognise the difference between a simplification and a replacement with simple ideas. A simplification is a description that isn’t comprehensive, but is still true. Replacing a complex idea with simple one, on the other hand, often boils down to telling a loosely related story for the sake of entertainment.

Abstraction is essential to scientific inquiry, and often analogy for the sake of one’s own understanding or that of an audience can be a tricky thing to grapple with. All too often when scientists and science writers try to convey especially tricky ideas they end up deviating from their premises and consequently end up communicating something almost wholly different than what they intend to describe.

Over the last few weeks I pointed out a few examples of “exceptional” analogy in science writing. Attentive readers may have noticed they were all sort of… not good. They all fall a bit flat for purposes of communicating the science behind them.. A tough task to be sure, popularisers of science have to balance accuracy against confusing their audience with esoteric nonsense. Wasn’t it Isaac Asimov that said any sufficiently specialised language is indistinguishable from rampant babble? Hopefully you enjoyed the cartoons.

Every single time

This one gets used quite often, and typically will form the pinnacle of a long line of increasingly wayward analogies. Space is mind-bogglingly big (much larger than the distance to the chemist’s, thanks D.A.) and also quite weird.

Don't actually stick your hand in the LHC

I have heard this one from some very clever people. Presumably particle physicists all switched to the rock-star analogy after they grew tired of watching listeners eyes glaze over when they delve into maths.

Mitochondria-The engines of life

Internal combustion in the cell

This one might be the closest to the mark. The engine/mitochondria analogy gets the point across that metabolism involves trade-offs, but does very little to convey a better understanding of the underlying mechanisms.

Philaephilia?

Philaephilia n. Temporary obsession with logistically important and risky stage of scientific endeavour and cometary rendezvous.

Don’t worry, the condition is entirely transient

Rivalling the 7 minutes of terror as NASA’s Curiosity rover entered the Martian atmosphere, Philae’s descent onto comet 67P/Churyumov-Gerasimenko Wednesday as part of the European Space Agency’s Rosetta mission had the world excited about space again.

Comets don’t have the classic appeal of planets like Mars. The high visibility of Mars missions and moon shots has roots in visions of a Mars covered in seasonal vegetation and full of sexy humans dressed in scraps of leather, and little else. But comets may be much better targets in terms of the scientific benefits. Comets are thought to have added water to early Earth, after the young sun had blasted the substance out to the far reaches of the solar system beyond the realm of the rocky planets. Of course, comets are also of interest for pure novelty: until Philae, humans had never put a machine down on a comet gently. Now the feat has been accomplished three times, albeit a bit awkwardly, with all science instruments surviving two slow bounces and an unplanned landing site. Unfortunate that Philae is limited to only 1.5 hours of sunlight per 12 hour day, but there is some possibility that a last-minute attitude adjustment may have arranged the solar panels a bit more fortuitously.

So if Rosetta’s Philae lander bounced twice, rather than grappling the surface as intended, and landed in a wayward orientation where its solar panels are limited to only 12.5% of nominal sun exposure, how is the mission considered a success?

Most likely, the full significance of the data relayed from Philae via Rosetta will take several months of analysis to uncover. Perhaps some of the experiments will be wholly inconclusive and observational, neither confirming nor denying hypotheses of characteristic structure of comets. For example, it seems unlikely that the MUPUS instrument (i.e. cosmic drill) managed to penetrate a meaningful distance into the comet, and we probably won’t gain much insight concerning the top layers of a comet beyond perhaps a centimetre or so. In contrast, CONSERT may yield unprecedented observations about the interior makeup of a comet.

In science, failures and negative findings are certainly more conclusive, and arguably more preferable, than so-called positive results, despite the selective pressure for the latter in science careers and the lay press. An exception disproves the rule, but a finding in agreement with theory merely “fails to negate” said theory. For example, we now know better than to use nitrocellulose as a vacuum propellant. Lesson learned on that front.

In addition to a something-divided-by-nothing fold increase in knowledge about the specific scenario of attempting a soft landing on a comet, I’d suggest we now know a bit more about the value of autonomy in expeditions where the beck-and-call from mission control to operations obviates real time feedback. Perhaps if Philae had been optimised for adaptability, it would have been able to maintain orientation to the comet surface and give Rosetta and scientists at home a better idea of its (final) resting place after detecting that the touchdown and grapple didn’t go through. Space science is necessarily cautious, but adaptive neural networks and other alternative avenues may prove useful in future missions.

I’ll eagerly await the aftermath, when the experimental and the telemetry data have been further analysed. The kind of space mission where a landing sequence can omit a major step and still have operational success of all scientific instruments on board is the kind of mission that space agencies should focus on. The Rosetta/Philae mission combined key elements of novelty (first soft landing and persistent orbiting of a comet) low cost (comparable to a fewspace shuttle missions), and robustness (grapples didn’t fire, comet bounced and got lost, science still occurred). Perhaps we’ll see continued ventures from international space agencies into novel, science-driven expeditions. Remember, the first scientist on the moon was on the (so far) final manned mission to Luna. Missions in the style of Rosetta may be more effective and valuable on all three of the above points, and are definitely more fundamental in terms of science achieved, than continuous returns to Mars and pushes for manned missions. In a perfect world where space agencies operate in a non-zero sum funding situation along with all the other major challenges faced by human society, we would pursue them all. But realistically, Philae has shown that not only do alternative missions potentially offer more for us to learn in terms ofscience and engineering, but can also enrapture the population in a transcendent endeavour. Don’t stop following the clever madness of humans pursuing their fundamental nature of exploring the universe they live in.

Why is there no confidence in science journalism?

UncertainScienceStoriesMS1939Small

Living in the so-called anthropocene, meaningful participation in humanity’s trajectory requires scientific literacy. This requirement is a necessity at the population level, it is not enough for a small proportion of select individuals to develop this expertise, applying them only to the avenues of their own interest. Rather, a general understanding and use of the scientific method in forming actionable ideas for modern problems is a requisite for a public capable of steering policy along a survivable route. As an added benefit, scientific literacy produces a rarely avoided side-effect of knowing one or two things for certain, and touching upon the numinous of the universe.

Statistical literacy is a necessary foundation for building scientific literacy. Widespread confusion about the meaning of such terms as “statistical significance” (compounded by non-standard usage of the term “significance” on its own) abounds, resulting in little to no transferability of the import of these concepts when scientific results are described in mainstream publications. What’s worse, this results in a jaded public knowing just enough to twist the jargon of science to support their own predetermined, potentially dangerous, conclusions (e.g. because scientific theories can be refuted by evidence to the contrary, a given theory, no matter the level of support by existing data, can be ignored when forming personal and policy decisions).

I posit that a fair amount of the responsibility for improving the state of non-specialist scientific literacy lies with science journalists at all scales. The most popular science-branded media does little to nothing in imparting a sense of the scientific method, the context and contribution of published experiments, and the meaning of statistics underlying the claims. I suggest that a standardisation of language for describing scientific results is warranted, so that results and concepts can be communicated in an intuitive manner without resorting to condescension, as well as conferring the quantitative, comparable values used to form scientific conclusions.

A good place to start (though certainly not perfect) is the uncertainty guidance put out by the Intergovernmental Panel on Climate Change (IPCC). The IPCC reports benefit from translating statistical concepts of confidence and likelihood into intuitive terms without sacrificing the underlying quantitative meaning (mostly). In the IPCC AR5 report guidance on addressing uncertainty [pdf], likelihood statements of probability are standardised as follows:

LikelihoodIPCCguidance

In the fourth assessment report (AR4), the guidance [pdf] roughly calibrated confidence statements to a chance of being correct. I’ve written the guidance here in terms of p-values, or the chance that results are due to coincidence (p = 0.10 = 10% chance), but statistical tests producing other measurements of confidence were also covered.

confStandardAR4

The description of results via their confidence rather than statistical significance, which is normally used, is probably more intuitive to most people. Few people in general readership readily discern between statistical significance, i.e. the results are likely to not be due to chance, and meaningful significance, i.e. the results matter in some way. Likewise, statistical significance statements are not even very well established in scientific literature and vary widely by field. That being said, the IPCC’s AR4 guidance threshold for very high confidence is quite low. Many scientific results are only considered reportable at a p-value of less than 0.05, or 5% chance of being an experimental artifact in the data due to coincidence, whereas the AR4 guidance links a statement of very high confidence to anything with less than a 10% chance of being wrong. Likewise, a 5-in-10 chance of being correct hardly merits a statement of medium confidence in my opinion. Despite these limitations, I think the guidance should have been merely updated to better reflect the statistical reality of confidenceand it was a mistake for the guidance for AR5 to switch to purely qualitative standards for conveying confidence based on the table below, with highest confidence in the top right and lowest confidence in the bottom left.

confAR5Standards

Adoption (and adaptation) of standards like these in regular usage by journalist could do a lot to better the communication of science to a general readership. This would normalise field-variable technical jargon (e.g. sigma significance values in particle physics, p-values in biology) and reduce the need for daft analogies. Results described in this way would be amenable to meaningful comparison by generally interested but non-specialist audiences, while those with a little practice in statistics won’t be any less informed by dumbing-down the meaning.

Edited 2016/06/25 for a better title, added comic graphic. Source for file of cover design by Norman Saunders (Public Domain)
23 Aug. 2014: typo in first paragraph corrected:

. . . meaningful participation in participating in humanity’s trajectory. . .

References:

Michael D. Mastrandrea et al. Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. IPCC Cross-Working Group Meeting on Consistent Treatment of Uncertainties. Jasper Ridge, CA, USA 6-7 July 2010. <http://www.ipcc.ch/pdf/supporting-material/uncertainty-guidance-note.pdf&gt;

IPCC. Guidance Notes for Lead Authors of the IPCC Fourth Assessment Report on Addressing Uncertainties. July 2005. <https://www.ipcc-wg1.unibe.ch/publications/supportingmaterial/uncertainty-guidance-note.pdf&gt;