Aiming for gold—improving reproducibility in hydrology studies

In six well-regarded hydrology and water resources journals published in 2017, the estimated percentage of studies whose results could be fully reproduced was only between 0.06 and 6.8 percent. This low level of reproducibility is not uncommon in hydrology studies—a fact many scientists readily acknowledge. However, a team of researchers at Utah State University may have found a solution to make these studies more reproducible.

In their paper, "Assessing data availability and research reproducibility in hydrology and water resources," published Feb. 26 in Nature's Scientific Data, David Rosenberg co-authors developed an online survey tool to assess the reproducibility of published research. The team reviewed 360 articles from six water resource journals published in 2017. Out of 360 articles, they could only fully reproduce results from four articles.

"Our survey tool breaks down the concept of scientific reproducibility into specific components of data availability, reproducibility of results and replicability of findings," said Rosenberg, an associate professor of civil and environmental engineering at USU. "We then suggest how authors, journals, funders and institutions can use the survey tool to increase low rates of reproducibility."

The authors say reproducibility can be broken down into three components:

• Are the data, models, code, directions for use and other artifacts used in the work available?
• Can artifacts be used to reproduce published results?
• Can the findings be replicated with new datasets?

The team's online survey tool consists of 15 questions and provides a checklist of the essential items needed for artifact availability and result reproducibility. Artifacts is an umbrella term for all data, software, models, code, directions and other materials needed to reproduce results within a study.

The team found that about 70 percent of the sampled articles stated some materials were available but only around 48 percent of the materials could be accessed online. Only around six percent of the sampled articles made artifacts publicly available, and only one percent of sampled articles made artifacts available and could be fully reproduced.

The authors said many articles were missing directions to generate results. If authors provide directions, they say, the number of articles that could be tested for reproducibility would double. Articles that made all artifacts available had a six in 10 chance of having some or all of their results reproduced. Two journals surveyed by the team required articles to state how artifacts can be accessed, and four journals encouraged statements. No journals required authors to make all artifacts available.

The survey tool can help recognize and encourage authors to achieve certain reproducibility levels. For example, authors can use the survey tool to self-assess the reproducibility of their results. Rosenberg and his team also recommend a medal system to recognize different levels of reproducibility:

• Bronze Medal: All artifacts are made available within the article or in open repositories
• Silver Medal: All artifacts are made available and results are fully reproducible
• Gold Medal: Results are fully reproducible and overall findings can be replicated in different settings with the same or different artifacts

Rosenberg recommend that medal icons be posted next to online articles to recognize authors for their reproducibility work and make it easier for readers to find top reproducibility practices. Rosenberg and his team assigned four silver medals and six bronze medals out of the 360 articles they reviewed. Awarding gold medals for replicability of findings remains an important line of future work.

"We hope the survey tool will help nudge authors, journals, funders and institutions to make scientific work more reproducible," said Rosenberg. "We welcome discussion to improve the survey tool and to improve the reproducibility of our science."