Functionality

The software aims at comparing ocean model output, which may be both physical or biogeochemical, to oceanographic observations.

The user selects

• a data set containing a pre-defined list of oceanographic stations, i.e., all stations of a particular measurement network,
• an oceanographic station from this dataset, either via a drop-down list or via a map,
• a variable to compare between model results and measurements, e.g., temperature,
• between one and four models whose results will be compared to the observed data, and
• a plot type of their choice.

Five different kinds of plots can be selected, see Figure 1:

• Time series
  • Model data are averaged vertically over a chosen depth interval and drawn as curves over time. Measurements from within the depth range are drawn as points.
• Vertical profile
  • Model data are time-averaged over a selected period and month range, e.g., Dec-Feb, and drawn as curves over depth. Individual measurements are drawn as points.
• Scatter plot
  • Model results at the same time and location as the measurements are plotted against these observations.
• Taylor diagram
  • The standard deviation in model data and measurements, the correlation between them and the standard error are shown in the same diagram following [1].
• Seasonal and/or long-term signal extracted with a Generalised Additive Mixed Model (GAMM)
  • Model data are vertically averaged to create a time series. Then, a GAMM model [2] is used to fit smooth splines through measurements and model results. Both long-term and periodic nal splines can be selected. The latter allows for a comparison of the seasonal cycle between model and measurements. This can be used to compare seasonality and long-term trends in the data.

A set of additional options exists depending on the plot type, e.g., showing the 5^th and 95^th percentile in a vertical profile plot. Also, the user can select a sub-range of time or depth, which allows a detailed exploration of individual aspects of model behaviour.

Implementation overview

The software is implemented as an R Shiny app [3], making it easy to use via an interactive web-based front-end. The back-end can run both on a private computer or on a server. It requires RStudio or Shiny Server, respectively, and a Linux operating system, possibly in a virtual machine. A Dockerfile is supplied such that the software can be used with example data without the need to install the software itself. The use of the Dockerfile and installation instructions are explained in the user’s guide. The software can load model data from a predefined list of models given in a text file. The same applies for measurement datasets. The app requires read access to the data over the (server-side) file system, an online upload of data is not intended. Model data have to be provided in the NetCDF format [4] and measurement data as text files or via an SQL database. We refer to the user’s guide for details on the format of the required data.

Alternatives

This software is, to our best knowledge, the first example of a web-based open-source solution aimed at validating ocean model output. There are, however, alternative software packages which could be used to obtain similar validation results.

The first alternative is the use of visualisation software for geodata. This includes the programs NOAA Ferret [5] and GrADS [6], which are specialised software solutions for geodata visualisation. The diversity of statistics and plots which can be produced by these programs strongly exceeds the possibilities in our software. This, however, comes at the expense of a dedicated script language which needs to be learned in order to perform the required tasks.

A second alternative is the use of more general script languages like R [7] or MATLAB [8], for which packages for model data access or validation routines exist. An example for accessing model results from the R programming language is the package rNOMADS [9]. An example for a library of scripts for doing a numerous set of validation plots is the open-source Python package VACUMM by Ifremer [10].

A third alternative to our Validator is the use of software which aims at assigning model scores as a measure of the goodness of the fit, rather than providing graphical evaluation. Software of this kind can be used for model intercomparison and assessing which model performs best compared to a predefined set of measurements. Examples for these tools are the NOAA model skill assessment software [11] or the ESMValTool [12], which is designed for obtaining performance metrics of Earth system models from the Coupled Model Intercomparison Project (CMIP6).

Checking the consistency of input arguments

The plotting process only continues if the input passes some basic checks, e.g., if the input type is numeric or if upper range limits are below lower range limits.

Loading model data

The required time and depth range of the selected variable is read from the input file and loaded into memory, for up to four selected models. Model data need to be provided as one NetCDF file per oceanographic station. For format details we refer to the user’s guide.

Loading measurement data

Measurement data can be either loaded from an ASCII file or from a database. Both methods are illustrated in the example setup we provide. We use the open-source data base MariaDB, but any database which supports SQL queries should work. R connects to the data base via the package RMariaDB. Two tables need to be provided in the data base: The “measurements” table contains the observation values and the time and depth in which they were taken, while the “variables” table contains details about the observed variable, e.g., the name and unit. Details can be found in the user’s guide. After the data are loaded from a data base, they are stored in a cache directory to minimise data base requests. In summary, this takes the following steps:

1. If data for the selected station and temporal range do not exist in the cache directory, perform a corresponding data base request and save the acquired data in ASCII format. During the processing of the data base request, the process cannot be aborted when the user selects a new plot, but its creation will be delayed.
2. Data will be loaded from the corresponding ASCII file in the cache directory.
3. The selected subrange of depth and time will be selected.

Select seasonal range

The user can select a range of months (e.g., Dec to Feb) which they want to include in the plot output. Both measurements and model data outside the given range will be discarded.

Calling the actual plot function, depending on the plot type

The actual plot function will create a plot showing both the model data (of up to four models) and observations. The “ggplot2” package is used. The first three plot types (time series, vertical profile, scatter plot) require no further explanation.

Choosing the option “Taylor diagram” will plot a Taylor diagram [1] displaying three measures of the model fit to the measurements at the same time: Pearson’s correlation coefficient, root mean squared error and the standard deviation.

Choosing the option “seasonal/long term trend” will fit each one Generalised Additive Mixed Model (GAMM) [2] through the observations and every model’s data. The data are modelled by fitting two smooth splines: One of them is periodic with a one-year period describing the seasonality of the measured variable, the second one describes the long-term behaviour. The random error terms are represented by a first-order autoregressive model (continuous AR-1) [2]. Plotting the resulting splines allows to independently evaluate whether or not (a) the seasonality of the observed variable and (b) the (non-linear) long-term trend match between model(s) and observations.

Archive

• Name: IOW institutional repository
• Persistent identifier: DOI: 10.12754/prog-2019-0001-01
• Licence: MIT
• Publisher: Hagen Radtke
• Version published: 1.0.1
• Date published: 16/05/19

Code repository

• Name: GitHub
• Identifier: https://github.com/hagenradtke/validator
• Licence: MIT
• Date published: 16/05/19