A Skeptic Over Coffee #1: Starter Kit


It takes effort and maintained vigilance to become an effective skeptic, with the penetrating mental focus to cut through the misleading. Honing one’s questioning acuity means hardening one’s mental defenses against charlatans, fraudsters, and the merely incompetent in all walks of life. With practice it’s possible to be the infamous “Reviewer Number 3” who gradually gets fewer and fewer invitations to provide peer-review for “paradigm shifting” articles from editors of high-impact journals. It may seem like a grandiose dream, but you too can in fact be the colleague who corrects the university press office’s outlandish claims about their own paper, causing their tenure review to be shelfed for another year (for failure to be interviewed on Science Friday</a<). If this glamourous lifestyle of modest claims and bold negations sounds appealing, read on!

I invite you to join me every once in a while to practice skepticism in these short segments designed to provide about one coffee's worth of skeptical inquiry. My day job pushing things around with lasers both takes a lot of time and requires that I drink a tremendous amount of coffee, so the concise aSOC format should fit right in with my new lab-monkey lifestyle.

Here is your Beginning Skeptics’ reading list:

  • A seminal paper by John Ioannidis runs the numbers on an over-abundance of false-positives in the scientific literature.
    John P.A. Ionnidis. Why Most Published Research Findings Are False. PLOS. (2005). DOI: 10.1371/journal.pmed.0020124

  • Retraction Watch is an important resource for any skeptic. If someone consistently publishes retractable articles and no one notices, does anyone lose their scientist licence?
  • Jeffrey Beall runs black lists of predatory publishers and journals taking advantage of pay-for-publish open access models atScholarly Open Access. Also consider John Bohannon’s misleading report generalising predatory practices by OA publishers and ensuing criticism of his approach.
  • And remember your statistics:
    Why it Always Pays to Think Twice About Your Statistics
    An investigation of the false discovery rate and the misinterpretation of p-values

  • UPDATE: Recent, interesting consideration of widespread inflation of scientific results.
    Megan L. Head, Luke Holman, Rob Lanfear, Andrew T. Kahn, Michael D. Jennions.
    The Extent and Consequences of P-Hacking in Science.
    PLOS. (2015) DOI: 10.1371/journal.pbio.1002106
  • Rubbish in, Garbage Out?

    Extraordinary claims require extraordinary press releases?

    You have probably read a headline in the past few weeks stating that NASA has verified that an infamous, seemingly reactionless propulsion drive does in fact produce force. You also might not have read the technical report that spurred the media frenzy (relative to the amount of press coverage normally allocated to space propulsion research, anyway), instead relying on the media reports and their own contracted expert opinion. The twist is that it seems to be the case that no one else- excepting perhaps the participants of the conference it was presented at– has read it either, and this includes myself and likely the authors of almost any other material you find commenting on it. The reason is that the associated entry in the NASA Technical Reports Server only consists of an abstract.

    The current upswing of interest and associated speculation on the matter of this strange drive is eerily reminiscent of other recent \begin{sarcasm}groundbreaking discoveries\end{sarcasm}: FTL neutrinos measured by the OPERA experiment and the Arsenic Life bacterium from Mono Lake, California. Both were later refuted, some important people at OPERA ended up resigning, and the Arsenic Life paper continues to boost the impact factors of the authors and publisher as Science Magazine refuses to retract it. (current citations according to Google Scholar number more than 300).

    I would venture that the manner of disclosing the OPERA findings was done more responsibly than the Arsenic Life paper. Although both research teams made use of press releases to gain a broad audience for their findings (note this down in your lab notebook as “do not do” if you are a researcher), the OPERA findings were at the pre-publication stage and disclosed as an invitation to greater scrutiny of their instrumentation, while the arsenic life strategy was much less reserved. From the OPERA press release:

    The OPERA measurement is at odds with well-established laws of nature, though science frequently progresses by overthrowing the established paradigms. For this reason, many searches have been made for deviations from Einstein’s theory of relativity, so far not finding any such evidence. The strong constraints arising from these observations makes an interpretation of the OPERA measurement in terms of modification of Einstein’s theory unlikely, and give further strong reason to seek new independent measurements.

    Notice the description of the search for exceptions to Einstein’s relativity as ” . . . so far not finding any evidence. . .” That despite the data they are reporting doing exactly that if anomalous instrumentation could be ruled out. This was a plea for help, not a claim of triumph.

    On the contrary, the press seminar associated with the release of Felisa Wolfe-Simon et al.’s A bacterium that can grow by using arsenic instead of phosphorus issued no such caveats with their claims. Likewise it was readily apparent in the methods sections of their paper that the Arsenic Life team made no strong efforts to refute their own data (the principal aim of experimentation), and the review process at Science should probably have been made more rigorous than standard practice. It is perhaps repeated too often without consideration, but I’ll mention the late, great Carl Sagan’s assertion that “extraordinary claims require extraordinary evidence.” The OPERA team kept this in mind, while the Arsenic Life paper showed a strong preference to sweep under the carpet any due diligence in considering alternative explanations. Ultimately, the OPERA results were explained as an instrumentation error and the Arsenic Life discovery has been refuted in several independent follow-up experiments (i.e. [1][2]).

    Is propellant-less propulsion on par with Arsenic Life or FTL neutrinos in terms of communicating findings? In this case I would lean toward the latter: more of a search for instrumentation error than a claim of the discovery of Totally New Physics. The title of the tech report “Anomalous Thrust Production from an RF Test Device Measured on a Low-Thrust Torsion Pendulum” denotes the minimum requisite dose of skepticism.

    Background reading below, but by far the best take on the subject is xkcd number 1404. The alt-text: “I don’t understand the things you do, and you may therefore represent an interaction with the quantum vacuum virtual plasma.”

    23/08/2014 several typos corrected
    [UPDATE Full version of tech report: http://rghost.net/57230791%5D via comments from http://ow.ly/ADJqb .
    http://www.wired.co.uk/news/archive/2014-07/31/nasa-validates-impossible-space-drive .
    http://www.wired.co.uk/news/archive/2014-08/07/10-qs-about-nasa-impossible-drive .
    http://www.wired.com/2014/08/why-nasas-physics-defying-space-engine-is-probably-bogus/?mbid=social_twitter .
    http://en.wikipedia.org/wiki/Quantum_vacuum_plasma_thruster .
    http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20140006052.pdf .
    http://www.wired.co.uk/news/archive/2013-02/06/emdrive-and-cold-fusion .
    http://www.aiaa.org/EventDetail.aspx?id=18582 .