SimEarth

The Earth—with its myriad shifting atmospheric, oceanic, land and ice components—presents an extraordinarily complex system to simulate using computer models.

But a new Earth modeling system, the Energy Exascale Earth System Model (E3SM), is now able to capture and simulate all these components together. Released on April 23, after four years of development, E3SM features weather-scale resolution — i.e., enough detail to capture fronts, storms and hurricanes — and uses advanced computers to simulate aspects of the Earth's variability. The system can help researchers anticipate decadal-scale changes that could influence the U.S. energy sector in years to come.

"With this new system, we'll be able to more realistically simulate the present, which gives us more confidence to simulate the future," says David Bader, computational scientist at Lawrence Livermore National Laboratory and overall E3SM project lead.

The E3SM project is supported by the U.S. Department of Energy's (DOE) Office of Biological and Environmental Research. "One of E3SM's purposes is to help ensure that DOE's climate mission can be met—including on future exascale systems," said Robert Jacob, a computational climate scientist in the Environmental Science division of DOE's Argonne National Laboratory and one of 15 project co-leaders.

To support this mission, the project's goal is to develop an Earth system model that increases prediction reliability. This objective has historically been limited by constraints in computing technologies and uncertainties in theory and observations. Enhancing prediction reliability requires advances on two frontiers: (1) improved simulation of Earth system processes by developing new models of physical processes, increasing model resolution and enhancing computational performance; and (2) representing the two-way interactions between human activities and natural processes more realistically, especially where these interactions affect U.S. energy needs.

"This model adds a much more complete representation between interactions of the energy system and the Earth system," said David Bader, a computational scientist at Lawrence Livermore National Laboratory and overall E3SM project lead. "With this new system, we'll be able to more realistically simulate the present, which gives us more confidence to simulate the future."

The long view

Simulating the Earth involves solving approximations of physical, chemical and biological governing equations on spatial grids at the highest resolutions possible.

In fact, increasing the number of Earth-system days simulated per day of computing time at varying levels of resolution is so important that it is a prerequisite for achieving the E3SM project goal. The new release can simulate 10 years of the Earth system in one day at low resolution or one year of the Earth system at high resolution in one day (a sample movie is available at the project website). The goal is for E3SM to support simulation of five years of the Earth system on a single computing day at its highest possible resolution by 2021.

This objective underscores the project's heavy emphasis on both performance and infrastructure—two key areas of strength for Argonne. "Our researchers have been active in ensuring that the model performs well with many threads," said Jacob, who will lead the infrastructure group in Phase II, which—with E3SM's initial release—starts on July 1. Singling out the threading expertise of performance engineer Azamat Mametjanov of Argonne's Mathematics and Computer Science division, Jacob continued: "We've been running and testing on Theta, our new 10-petaflop system at Argonne's Leadership Computing Facility, and will conduct some of the high-res simulations on that platform."

Researchers using the E3SM can employ variable resolution on all model components (atmosphere, ocean, land, ice), allowing them to focus computing power on fine-scale processes in different regions. The software uses advanced mesh-designs that smoothly taper the grid-scale from the coarser outer region to the more refined region.

Adapting for exascale

E3SM's developers—more than 100 scientists and software engineers—have a longer-term aim: to use the exascale machines that the DOE Advanced Scientific Computing Research Office expects to procure over the next five years. Thus, E3SM development is proceeding in tandem with the Exascale Computing Initiative. (Exascale refers to a computing system capable of carrying out a billion [10¹⁸] calculations per second—a thousand-fold increase in performance over the most advanced computers from a decade ago.)

Another key focus will be on software engineering, which includes all of the processes for developing the model; designing the tests; and developing the required infrastructure, including input/output libraries and software for coupling the models. E3SM uses Argonne's Model Coupling Toolkit (MCT), as do other leading climate models (e.g., Community Earth System Model [CESM]) to couple the atmosphere, ocean and other submodels. (A new version of MCT [2.10] was released along with E3SM.)

Additional Argonne-specific contributions in Phase II will center on:

• Crop modeling: Efforts will focus on better emulating crops such as corn, wheat and soybeans, which will improve simulated influences of crops on carbon, nutrient, energy and water cycles, as well as capturing the implications of human-Earth system interactions
• Dust and aerosols: These play a major role in the atmosphere, radiation and clouds, as well as various chemical cycles.

Collaboration among - and beyond - national laboratories

The E3SM project has involved researchers at multiple DOE laboratories including Argonne, Brookhaven, Lawrence Livermore, Lawrence Berkeley, Los Alamos, Oak Ridge, Pacific Northwest and Sandia national laboratories, as well as several universities.

The project also benefits from collaboration within DOE, including with the Exascale Computing Project and programs in Scientific Discovery through Advanced Computing, Climate Model Development and Validation, Atmospheric Radiation Measurement, Program for Climate Model Diagnosis and Intercomparison, International Land Model Benchmarking Project, Community Earth System Model and Next-Generation Ecosystem Experiments for the Arctic and the Tropics.