The numerical global atmosphere-chemistry model EMAC (ECHAM/MESSy Atmospheric Chemistry) is a modular application used for climate modeling simulations . EMAC uses the chemical kinetic module MECCA , utilizing the Kinetic Pre-Processor (KPP) general purpose open-source software tool  to calculate the concentrations and the interactions between different chemical species in the atmosphere. Numerically, solving atmospheric chemical kinetics is one of the most computationally intensive tasks in atmospheric chemical transport simulations.
The MECCA sub-model uses KPP to numerically solve ordinary differential equations (ODE) describing atmospheric chemical kinetics. KPP takes as input chemical reactions written in a domain-specific language and produces C and FORTRAN compatible code. The output ODE solver allows for the temporal integration of the full kinetic system. KPP utilizes the sparsity of the Jacobian matrices to increase the efficiency of the solver .
In typical climate simulations, chemical kinetics can take up to 90% of execution time . To address this computational challenge, this paper presents a source-to-source parser that transforms the output of KPP from FORTRAN to GPU accelerated code by generating a CUDA  compatible solver . The goal is to significantly improve the performance of numerical chemical kinetics (in terms of time-to-solution and problem complexity) in climate simulation models using GPU accelerators.
The software provides a purpose-built acceleration pathway for EMAC, rather than the general closed source solution KPP-A , which generates GPU-accelerated code from KPP-language directly. There have also been similar efforts in the WRF-Chem model .
The implementation uses a source-to-source parser written in the Python programming language to generate a CUDA  compatible solver, by parsing the KPP preprocessor auto-generated FORTRAN code. Each GPU thread calculates the chemical concentrations of an individual cell. The solver uses the temporary arrays between different steps and they are allocated in the stack memory with the exception of the RCONST array that is stored in global memory. The memory required for temporary arrays depends on the configuration of the chemistry and can go up to 50 KB per CUDA thread. The intermediate values are stored in temporary arrays using the double representation. The accelerated code uses 3 KB of shared memory and 0.5 KB of constant memory when indirect arrays are not used. The parser allocation additional constant memory up to 2 KB when indirect accesses are used. All the methods that are available in the KPP numerical library under MECCA are supported.
The computation data structures are subdivided in runtime-specified arrays of columns in the atmosphere, with the memory of each array transferred to the GPU global memory and each grid box calculated on a separate GPU core to achieve massive parallelization, as shown in Figure 1. The CUDA chemical kinetics solver comprises three steps, also presented diagrammatically as a flow chart in Figure 2:
There are two files required to enable the GPU utilization: i) f2c_alpha.py and ii) kpp_integrate_cuda_prototype.cu. The pre-processor is executed by running python f2c_alpha.py in the messy/util directory.
When offloading to GPUs, the number of cells must not exceed 12288. The application calculates the number of cells by multiplying the number of columns by the number of levels for the atmosphere. The user can specify the number of columns by using the NVL (NPROMA) runtime parameter in the configuration of the EMAC.
To ensure the quality of the code, we conduct unit testing by comparing the GPU accelerated with a pure Fortran simulation for one model year, using 155 species and 310 reactions. We compare the output of chemical element concentrations between the CPU only and accelerated version after the first time-step to calculate the relative error. The results show a median difference of 0.00005% with the maximum difference value of 0.54% using identical inputs, ensuring the high accuracy of the chemical solver. This is well within the accuracy criterion, asserting the numerical correctness of the GPU kernel.
Finally, we compare the results of aggregated mass of the CPU-only and GPU-accelerated version, over one year of simulated time. This test aggregates the error created over time using the L1-norm method and investigates the stability of the model. The results show that the median value of the difference in aggregated mass is less than 5%, well within the expected margin of differences stemming from architecture and compiler implementations.
The software is compatible with any operating system that supports the CUDA SDK.
The script is written in the Python language and it has been tested with Python version 2.7. The script produces code that supports the the CUDA programming model version 2.0 or later.
Each CPU process that offloads to GPU requires a chunk of the GPU VRAM memory, whose size is dependent on the number of species and reaction constants in the MECCA mechanism. The number of GPUs per node and VRAM memory available in each GPU dictates the total number of CPU cores that can run simultaneously. We strongly recommend that at least 2.5 GB of VRAM per CPU process should be available on each GPU. Note that if not enough memory is available, the CUDA runtime will silently fail – without any warning.
The script requires Python version 2.6 or 2.7. The source code produced requires the CUDA SDK 6.5 or newer.
Name: MECCA – KPP Fortran to CUDA source-to-source pre-processor.
Persistent identifier: DOI: 10.5281/zenodo.546811
Licence: MIT License.
Publisher: The Cyprus Institute.
Version published: 1.1.
Date published: 18/04/17.
Licence: MIT License.
Date published: 12/12/16.
The source-to-source parser core can be used by other researchers to transform FORTRAN 90 code to CUDA accelerated code. The methodology can be used as template to create high performance computing capabilities on GPUs for scientific code that is currently written in FORTRAN 90.
The CUDA compatible ODE solvers comprises the Runge Kutta methods and the linearly-implicit Rosenbrock family of solvers, and may be used for chemical kinetics and numerical integration outside the field. The software is supported by the developers. Any inquiries should be addressed by e-mail.
The authors would like to thank Patrick Jöckel for his support during the packaging and integration of the source-to-source parser and Klaus Klingmueller for the graphics in Figure 1.
The authors have no competing interests to declare.
Christou, M, Christoudias, T, Morillo, J, Alvarez, D and Merx, H (2016). ‘Earth system modelling on system-level heterogeneous architectures: EMAC (version 2.42) on the Dynamical Exascale Entry Platform (DEEP)’. Geoscientific Model Development 9(9): 3483.DOI: https://doi.org/10.5194/gmd-9-3483-2016
Christoudias, T and Alvanos, M (2016). Accelerated chemical kinetics in the EMAC chemistry-climate model In: ‘High Performance Computing & Simulation (HPCS), 2016 International Conference on’. IEEE, pp. 886–889, DOI: https://doi.org/10.1109/hpcsim.2016.7568427
Damian, V, Sandu, A, Damian, M, Potra, F and Carmichael, G R (2002). ‘The kinetic preprocessor KPP – a software environment for solving chemical kinetics’. Computers & Chemical Engineering 26(11): 1567–1579, DOI: https://doi.org/10.1016/S0098-1354(02)00128-X
Jöckel, P, Kerkweg, A, Pozzer, A, Sander, R, Tost, H, Riede, H, Baumgaertner, A, Gromov, S and Kern, B (2010). ‘Development cycle 2 of the modular earth submodel system (messy2)’. Geoscientific Model Development 3(2): 717–752, DOI: https://doi.org/10.5194/gmd-3-717-2010
Linford, J C, Michalakes, J, Vachharajani, M and Sandu, A (2009). Multi-core acceleration of chemical kinetics for simulation and prediction In: Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis. ACM, p. 7.DOI: https://doi.org/10.1145/1654059.1654067
Sander, R, Baumgaertner, A, Gromov, S, Harder, H, Jöckel, P, Kerkweg, A, Kubistin, D, Regelin, E, Riede, H, Sandu, A, Taraborrelli, D, Tost, H and Xie, Z-Q (2011). ‘The atmospheric chemistry box model CAABA/MECCA-3.0’. Geoscientific Model Development 4(2): 373–380, DOI: https://doi.org/10.5194/gmd-4-373-2011
Zhang, H, Linford, J C, Sandu, A and Sander, R (2011). ‘Chemical mechanism solvers in air quality models’. Atmosphere 2(3): 510–532, DOI: https://doi.org/10.3390/atmos2030510