Multi-disciplinary Research at Work: Combustion Exascale Co-Design Center Established

Combustion research has never been timelier; in an era when resources are limited, energy costs are rising, and environmental concerns demand consideration, top researchers must find new ways to maximize efficiency while minimizing costs. Combustion accounts for roughly 85% of the total energy used in the U.S.–developing technologies to improve combustion efficiency necessitates a deeper understanding of different fuels and combustion systems under a variety of complex conditions. One method researchers currently use to study combustion is direct numerical simulation (DNS) in conjunction with high performance computing; however, as simulations increase in complexity and reach towards practically relevant thermochemical and flow conditions, an entirely different type of computing system is needed to keep up with the volume of calculations and data generated.

In the fall of 2011, Sandian Dr. Jackie Chen and Dr. John Bell of Lawrence Berkeley National Laboratory were awarded a 5-year-grant from the Department of Energy (DOE) to establish a combustion co-design center of interdisciplinary researchers–computer scientists, mathematicians and computational combustion scientists. This collaboration, known as the Center for Exascale Simulation of Combustion in Turbulence (ExaCT), also includes researchers from Lawrence Livermore, Lawrence Berkeley, Oak Ridge and Los Alamos National Laboratories, as well as the National Renewable Energy Laboratory, the University of Texas at Austin, the University of Utah, Stanford University, Georgia Tech and Rutgers University. Application-driven co-design is a process where scientific requirements influence architectural design, and technology constraints inform the design of algorithms and software.  Through this tight-knit iterative loop, ExaCT researchers hope to contribute to the design of an exascale (10¹⁸ flops) computer system optimized to the critical applications of the DOE Office of Science in the 2018-2020 timeframe.  Combustion is one of these critical applications.

The Challenge

Current first principles combustion simulation codes run on petascale architectures, meaning the system performs one quadrillion, or 10¹⁵, calculations per second. Using these codes, researchers are able to simulate canonical turbulent flames of relatively simple fuels, such as hydrogen and small hydrocarbons, at atmospheric pressure. Although the simulations that describe the coupling between chemical kinetics and transport at atmospheric conditions provide a useful foundation, exascale architectures can help enable researchers to study increasingly complex intricacies of turbulence interactions in the high-pressure environments like those found in real world engines and gas turbines. The challenge is that newer, faster architectures cannot simply be scaled up from existing processors because such an extrapolation will lead to a machine that costs more to run than to build in the first place. Whereas once the only metric that was needed in a supercomputing system was flops, now energy consumption has surpassed flops as the foremost constraint.

The Solution

Due to power constraints, computing at the exascale will require extreme levels of parallelism – as much as billion way concurrency – requiring revolutionary software and hardware technologies that have to be optimized for performance, power, energy, and reliability. Applications co-design of the entire stack, including combustion applications, programming models, scientific data management and in situ analytics, runtime systems, and hardware is needed to design solution stacks that are dynamic, power efficient, as well as resilient to software and hardware faults.  A key part of ExaCT, therefore, is to tailor the system so that its strengths are tuned to needs unique to combustion research. This type of approach is intended to be highly iterative—as researchers design new architectures, proxy applications of key aspects of a combustion simulation will be run in an architectural simulator with an abstract machine model, representing potential characteristics of an exascale architecture, to see if a hypothetical system is well suited to study the scenario at hand.

ExaCT will use proxy applications (apps), which are smaller code fragments that represent an important functionality in a production code, to represent the combustion application workload and requirements to the exascale ecosystem.  Through architectural simulation of future exascale designs represented by an abstract machine model, the proxy apps will be used by both high performance computing vendors and the co-design team to understand the effects of hardware tradeoffs and to explore new programming languages and models, algorithms and technologies.  The proxy apps will evolve significantly through the iterative co-design process.

The power requirements of combustion codes depend on voltage and frequency; reducing both will require flat or reduced clock frequencies. Additionally, the energy cost of data movement may exceed the cost of computation, implying the need for stricter control of data locality in the newer architectures. Software codes for conducting simulations will also have to be designed to adapt to emerging hardware trends.  For example, in combustion codes, novel algorithms for numerical discretization of partial differential equations solving the reacting Navier-Stokes equations are required to express significantly greater concurrency, and potentially trade increased computational intensity for reduced data movement. To access relevant high pressure regimes of combustion, the methodology will combine high-order accurate discretization with adaptive mesh refinement.

Beyond simply performing an exascale combustion simulation, additional challenges arise in the end-to-end problem.  The traditional approach used in the combustion community for scientific data management, analysis and visualization is to periodically write synchronous checkpoints to persistent storage for restarts and analytics. This approach becomes infeasible at the exascale because I/O speeds will significantly lag behind compute speeds, and the total amount of data stored for post processing must be reduced even as the simulations get increasingly detailed. To address these technology constraints, ExaCT will develop the capability to analyze and visualize simulation results in situ as part of the simulation process by enabling combustion scientists to choose aspects of the simulation to measure and instrument the code to extract those measurements.  Co-design research in programming models, runtime systems, and resiliency are also needed to perform in situ data analytics.

Says Dr. Chen, “Successful co-design will enable exascale combustion simulations to advance the science base that will ultimately contribute to development of non-petroleum-based fuels and their efficient utilization, crucial to the goal of reducing US dependence on petroleum usage.”