Can Sigma Xi Fix the Reproducible Science Crisis: A Member Responds

Dear officers, staff and members, 

I would like to contribute to the current quarterly conversation about fixing the reproducible science crisis. I suggest this "crisis" should not be considered in the singular. Rather, it is but one aspect of a greater problem to be examined in toto. This is referenced in your introduction to this conversation as a "growing perception that science is not a worthwhile investment" and "...misunderstanding of the scientific process..."  It is often called simply, "The War on Science." Fixing "reproducibility" or even a few of the numerous contributors to the problem is placing Band-Aids on a gaping wound.

I argue the problem lies in communication from a particular branch of science to the public. This includes all aspects, from the initial report, to publications, and to press releases to columnists and science reporters. The information path is broken and pieces are strewn most evenly across the entire channel.

I have given a great deal of thought to this tragedy as I watched it evolve over the past many decades. I have made presentations to science reporters trying to assist them in their impossible task of culling the truths from the deluge of press releases and journal articles but have been unsuccessful in attempts to publish comments. This conversation promises a more receptive ear.

Ideas to Solve the Problem So Far

A brief background to frame my argument:

1. By way of solutions to the problem, Danny Kingsley suggests open research and praises registered reports. 

2. Jim Grange says murmurings of low reproducibility began in 2011, and blames pressures, Publish or Perish, etc. and also lauds registered reports. 

3. Ottoline Leyser gets closer to the root causes of the problem by summing it up as a publication crises and cites some real reasons for it. She suggests a reward system that might shift research publication emphasis to quality of results as opposed to quantity of "positive" results. She has an excellent synopsis of problems in her article in The Conversation, "The dark side of research – when chasing prestige becomes the prize."

4. There has been discussion of these and similar topics in various blogs but little written material. A noteworthy exception is Shawn Otto's recent 500 page treatise, The War on Science. He blames the demise of respect for science on special interests, operative in various forms since Galileo's time, with but a few passing comments related to any problems internal to science and its practitioners. As corrective measures, he suggests 15 "battle plans," all but one directed outward, away from science itself. Though he does push for a Scientific Code of Ethics, he appears to point his hoses at the smoke rather than the fire. Mr. Otto has done a lot of homework and his writing is compelling but to my mind he misses the point, as do all the blogs and journal articles I have read. Even those that come close focus on but one or two pieces of the broken channel.

Most of these suggestions or recommended changes in procedures are not new. Several authors in many journals have expressed concern about incomplete reporting, publication bias, routes to tenure, catchy headlines, poor reporting, etc. Note, I do not include fraud or cheating here; they are relatively minor, separate issues.

Though blatant fraud and cheating make dramatic headlines from time to time, the real damage comes from the public’s mindset—not from arguments over global warming or whether the "moon landings" were staged in Arizona but rather  "when you toy with my health and wellbeing, I pay attention." At least most of the public once did  and some still do. 

A History of the War on Science

Now, to get a better understanding of what has happened, consider the public view of science during two periods of fairly recent history: roughly mid-19th to mid-20th century and the mid-20th through current times.

That first period might be called a Golden Age of Science as viewed by the public eye. Such a view was indeed justified by the maturing machine age, life improving inventions and, to a great extent, the life-extending medical miracles (life expectancy nearly doubled) wrought in large part by epidemiologists of the era. Health talks and scientific expositions drew large crowds. The Snake Oil Liniment era was crumbling under new scientific methods and recommendations which the public saw to be working. For the better part of 100 years, science was lauded and trusted. That spirit opened funding for several major undertakings such as the Framingham Study in 1948 and the uncountable number that followed and continue today. Recommendations for foods, supplements, and physical activities poured forth. 

Perhaps due in large part to the problems cited by Leyser, the public was, and still is, bombarded with poorly based, conflicting, usually erroneous, and simply bad medical advice. Most of it is speculative (reproducible?) and much of it is simply wrong. Nonetheless, it is widely published and (diminishingly) read simply because it DOES affect "MY health and wellbeing."

The flood of increasingly bad reports began shortly after the Surgeon General's report on smoking. Researchers were eager to jump on the bandwagon; it meant funding and tenure. By 1995 the deluge had become so disgusting, E. Ross vented his anger in a Forbes magazine article: "Lies, Damned Lies and Medical Statistics." 

A few other examples: 

1. L.K Altman. Promises of Miracles: News Releases Go Where Journals Fear to Tread. The New York Times. 1995. 

2. M. Angell. Overdosing on Health Risks. New York Times Magazine. 1997

3. M. Angell. Drug Companies & Doctors: A Story of Corruption. The New York Review of Books. 2009.

Yet even today, syndicated medical columnists, posing as experts who should know better, continue to cite single studies, be they good, bad, or indifferent, to support their arguments. So what if they are contradicted next year? Isn't that part of the scientific process?

Given this torrent of ever-changing information about "MY" health and lifestyle, presented as fact, seen as some as "damned lies" but believed, tried, and failed by many, what can you expect by way of public support?

What Scientists and Publishers Can Do

Implementation of the recommendations cited earlier is certainly highly desirable. However, researchers dealing in any aspect of health related subjects, epidemiologists in particular, have a moral responsibility greatly exceeding that of all others. Their papers are the few that are widely circulated, consumed by anxious readers who are inevitably disappointed and ultimately disgusted with all of science.

This group of researchers must stop writing up partial results, avoiding disclaimers, omitting caveats, and commonly exaggerating only positive results.

Peer reviewers must actually do their job, question dubious methods, and demand realistic and honest reporting. 

And, yes, journal editors share a lot of the blame. One gets the impression they are competing with check-out counter tabloids. If they claim their journal to be "peer reviewed" all articles should be, and by responsible reviewers, not simply published by editor consensus in order to be first to go to press. Publication bias must stop. Negative results are as important to science as positive ones.

I experienced a striking example of public impression stemming from poor publication practices: In 1990, during casual conversation, I mentioned The New England Journal of Medicine. A colleague instantly spoke up with “Oh, you mean the National Enquirer of the medical world." Sure, one person, one comment, but it was spontaneous and reflected a deep-seated feeling. Perhaps unfortunate that he chose the NEJM, he could have cited any number of others.

No, the cure is anything but easy. A lot of bullet-biting is required but if scientists are really concerned about  "evidence ... being discounted and dismissed among ...  our population," the very unscientific activities of the past 55-plus years will have to cease. The entire communications channel needs an overhaul. Then, even if this were accomplished, reviving a good public image could take a generation or two. The current disgust was over 50 years in the making.

What Sigma Xi Can Do

If Sigma Xi is sincere in its professed objectives, it must undertake greater proactive measures. 

How about:

1. A "Certified Reviewers" group (with stringent entrance requirements, reviewed annually).  As opposed to reviewing submissions for publication, such a group would write critiques of published papers, good and bad, acting as a watchdog for the profession. These critiques should be written in concise layman language and might become syndicated,  or at least appear in major newspapers as letters, op-eds, or similar.
   
   

2. A Best Papers Club wherein membership requires "Excellence in Scientific Writing." It would be important to include journal editors and science writers as well as medical columnists in the pool of candidates. Broad publication of membership would help, perhaps as press releases.
   
   

3. Recognizing that Sigma Xi membership and its sphere of influence is limited, it could strongly encourage federal agencies (NIH, etc.) to increase their current efforts on improving reports or initiate them where they have not been started.
   
   

4. Recent collaboration efforts by Sigma Xi are a good start at expanding its influence and building muscle. These should continue.

The argument that "honor societies" do not do such things may now be passé as indicated by this "new initiative to promote and cultivate quarterly conversations about issues relevant to our members’ needs and interests."

Sincerely,
John Fennick
Sigma Xi Member Since 1960

Photo credit: Scientists investigating the plague. Wellcome Library, London. https://creativecommons.org/licenses/by/4.0/. Cropped image.