Introduction

In recent years decadal climate predictions have become more and more popular in the climate science community. Typically an ensemble of so-called hindcast experiments with starting dates between 1960 and today are performed enabling a verification against observational data. But the evaluation and validation of decadal climate prediction systems is both a scientific and a technical challenge in the current climate research.

There are several convincing arguments for using a standardized tool for decadal model evaluation. For instance, model development stages and interim test phases of (in this case decadal climate predictions) can be assessed easily. It also simplifies the comparison of decadal prediction systems developed by different modeling groups.

Goddard et al. [2] proposed a framework for verification of decadal hindcasts addressing two key questions:

1. Do the initial conditions in the hindcast lead to more accurate predictions of the climate, compared to uninitialized climate change projections?

2. Is the prediction model’s ensemble spread an appropriate representation of forecast uncertainty on average

The first question is addressed through the Murphy- Epstein decomposition of the Mean Squared Error Skill Score (MSESS) [3],[4] which is based on the Mean Squared Error and can be used to compare two different model outputs (e.g. initialized and uninitialized). The MSESS is a deterministic skill measure of the ensemble mean. For the second question Goddard et al. [2] suggest to use a modified version of the Continuous Ranked Probability Skill Score (CRPSS) [5], which compares the average ensemble spread with the mean squared error. We extended the probabilistic part with the logarithmic ensemble spread score [6], which is a direct estimate for the skill of the mean ensemble spread.

MurCSS follows this framework and offers a standardized and reproducible way to calculate the above mentioned metrics for decadal prediction systems including a bootstrap method to determine significance levels. A detailed explanation of the tool, its output, and a tutorial how to use the tool can be found on our web-page ( https://www-miklip.dkrz.de/about/murcss/ ).

Implementation and architecture

MurCSS is a command line tool written in Python [7] using the Python-CDO-Interface [8]. The plots are produced and saved using the python library Matplotlib [9].

The tool architecture can be separated into three components (see Fig. 1), file input/preparation, metric calculation, and plotting routines. The file input searches for valid files and prepares them for the actual metric calculation. This component makes use of the MiKlip file database which is based on the international standards CMOR [10] and NetCDF [11]. It is also possible using the simplified file input component (findFilesCustom.py) which can be easily adjusted to the local database structure. The main part is the metric calculation, where the actual skill score calculation takes place. Most calculations are performed using CDO or the NumPy package, applying multiprocessing whenever possible. After processing the results are stored in the common NetCDF data format. In the last step these files are visualized using plotting routines.

Quality control

We developed the following strategy to assure the quality of the system.

a) We assembled a set of test-data.

b) Unit tests, which were constantly extended as soon as new problems were observed.

c) Users within MiKlip helped as beta-testers to improve the tool.

Operating system

MurCSS has been developed and tested on Linux. Although the tool should run on every system where Python and CDO are installed.

Programming language

Python 2.7, CDO 1.5.4

Dependencies

• Python Modules:

• Matplotlib >= 1.1.0
• Basemap
• NumPy >= 1.5.0
• SciPy >= 0.8.0
• CDOpy

• CDO

• NetCDF Libraries

Archive

Code repository

Language

English