Parameter Description

The first step in our generalised workflow is a description of the model parameters and how they might vary in the sampling phase of the VVUQ pattern. Typically, the user will specify all numerical parameters, the distributions from which they should be drawn and physically acceptable limits on their values.

Campaign

EasyVVUQ workflows are coordinated by an object called a “Campaign”. This contains a common database, the “CampaignDB”, which contains information on the application(s) being analysed alongside the runs mandated by the sampling algorithm(s) employed. The “Campaign” handles all validation and transfers information between each stage of the workflow.

The run information stored in the “CampaignDB” includes a status flag which indicates where in the VVUQ workflow the run is. The recorded steps are when: (1) a set of parameters for a run are added to the database (NEW), (2) simulation inputs are generated (ENCODED) and (3) simulation output is successfully read and prepared for analysis (COLLATED).

Samplers

A “Sampler” populates the “CampaignDB” with a set of run specifications based on the parameter description provided by the user. Each “Sampler” is designed to employ one of a range of algorithms, such as the Monte Carlo or Quasi Monte Carlo approaches [16, 18]. They deal with generic information in the sense that all parameters use the nomenclature and units provided by the user rather than anything specific to any application or workflow.

Encoders

The role of an “Encoder” is to convert generic parameter descriptions into inputs (for example configuration files) which can be used in a specific application. Included in the base application is a simple templating system in which values are substituted into a text input file. For many applications it is envisioned that specific encoders will be needed and the framework of EasyVVUQ means that any class derived from a generic Encoder base class is picked up and may be used. This enables EasyVVUQ to be easily extended for new applications by experienced users.

Decoders

The role of a “Decoder” is twofold, to record simulation completion in the “CampaignDB” and to extract the output information from the simulation runs. Similarly to an “Encoder”, a “Decoder” is designed to be user extendable to facilitate analysis of a wide range of applications.

Collation

“Collation” elements gather “Decoder” output across multiple runs to provide a combined and generic expression of the simulation results for further analysis (for example, the default is to bring together output from all simulation runs in a pandas dataframe).

Analysis

The final goal of any VVUQ workflow is an analysis which provides information on the simulation output across a range of runs. Different types of analysis (for example bootstrapping of multiple runs from varied initial conditions) are, or will be, provided by EasyVVUQ.

Dataflow

Using these concepts, we may construct the generalized VVUQ workflow depicted in Figure 1 in terms of VVUQ elements, as shown in Figure 2. A database is populated with runs generated by the chosen sampling element. These generic run descriptions are encoded to their application specific input formats and executed. A collation element aggregates the output from finished runs, using the appropriate decoder to extract the desired information. Finally one or more analyses can be carried out on this aggregated data. More runs can subsequently be drawn from the sampler, executed, and collated into the same output for analysis.

The workflow is orchestrated via a Campaign object that stores information such as run descriptions, current status of a run in the workflow (NEW, ENCODED or COLLATED), and the collated output in its database, which we refer to as the CampaignDB. A diagram of the database structure is shown in Figure 3. Each run stored in the run table retains the id of the campaign to which it belongs, the application it is generated for, and the sampler which generated it. Runs may also be filtered by their currently recorded status. The collated data is also stored in the database, although this is not shown in Figure 3 as the table fields depend on the output being collated.

The ultimate goal for EasyVVUQ to support complex multiscale workflows has shaped the design of the Campaign database, which must facilitate storing information from multiple applications. In EasyVVUQ an “app” is defined as a set of parameters (and fixtures which are our term for data sources where the paths may need to be manipulated by Encoders), an Encoder and a Decoder.

Installation

EasyVVUQ is available on the Python Package Index (PyPI), and can be installed using

pip install easyvvuq

for python versions 3.6 and above, with documentation and tutorials provided at https://easyvvuq.readthedocs.io. The latest development version can be obtained directly from the git repository at github.com/UCL-CCS/EasyVVUQ.

Features

The EasyVVUQ library is designed to be easily extended but already implements a variety of UQ algorithms (frequently building upon those found in the chaospy library [4]). At present, Stochastic Collocation, Polynomial Chaos Expansion, Quasi MonteCarlo, and parameter sweeps are implemented as sampling elements, some with corresponding analysis elements. It is envisioned that novel algorithms will be designed and implemented within the VECMA project and these integrated with the library, for example semi-intrusive methods for multiscale applications [10].

Data processing and storage within EasyVVUQ is handled using well established libraries. pandas dataframes are used as a ‘standard’ container for collated data, allowing simpler interfaces to be designed for the Analysis elements. The package includes an integrated wrapper around pandas summary statistics, alongside bootstrap statistics and sampler coupled analysis functions for Stochastic Collocation and Polynomial Chaos Expansion workflows. Interaction with databases is via sqlalchemy [9] (www.sqlalchemy.org), which provides a choice of several database backends. The use of flexible technology means that, depending on the application and resource on which it is to be run, different database choices will provide the needed performance, scalability and availability.

One of the major motivations behind employing a database is to allow restarting of the workflow – an important consideration for HPC workflows involving a large number of runs, for which the cumulative computational cost may become very large. Entire Campaigns can be restarted from the contents of the database, in which VVUQ elements such as samplers serialize their state. At present this serialized state is a JSON format string. For more complex objects this is achieved using the jsonpickle (github.com/jsonpickle/jsonpickle) library.

Physical range and type checking is performed on all parameters using the cerberus python library (github.com/pyeve/cerberus).

Example application: Cooling coffee cup

In this section, we illustrate the intended EasyVVUQ v0.5 [14] workflow using the following basic example script, a python implementation of the cooling coffee cup model used in the uncertainpy documentation (code for which is in the tests/cooling/subdirectory of the EasyVVUQ distribution directory). The code takes a small key/value pair input and outputs a comma separated value (CSV) file. The model uses Newton’s law of cooling to evolve the temperature, T, over time (t) in an environment at T_env:

The constant κ characterizes the rate at which the coffee cup transfers heat to the environment. In this example we will analyze this model using the polynomial chaos expansion (PCE) UQ algorithm. We will use a constant initial temperature T₀ = 95°C, and vary κ and T_env, sampling them from a uniform distribution in the ranges 0.025–0.075 and 15–25 respectively.

Below we provide a commented script that shows how the Campaign is built up and then employed. We also provide an outline of how each element is set up:

import easyvvuq as uq
import chaospy as cp
# Set up a fresh campaign called "coffee_pce"
my_campaign = uq.Campaign(name='coffee_pce')
# Define parameter space
params = {
"temp_init": {"type": "float", "min": 0.0, "max": 100.0, "default": 95.0},
"kappa": {"type": "float", "min": 0.0, "max": 0.1, "default": 0.025},
"t_env": {"type": "float", "min": 0.0, "max": 40.0, "default": 15.0},
"out_file": {"type": "string", "default": "output.csv"}
}
# Create an encoder, decoder and collater for PCE test app
encoder = uq.encoders.GenericEncoder(
template_fname='cooling.template',
delimiter='$',
target_filename='cooling_in.json')
decoder = uq.decoders.SimpleCSV(target_filename="output.csv",
output_columns=["te"],
header=0)
collater = uq.collate.AggregateSamples(average=False)
# Add the app (automatically set as current app)
my_campaign.add_app(name="cooling",
            params=params,
            encoder=encoder,
            decoder=decoder,
            collater=collater)
# Create the sampler
vary = {
      "kappa": cp.Uniform(0.025, 0.075),
      "t_env": cp.Uniform(15, 25)
}
my_sampler = uq.sampling.PCESampler(vary=vary, polynomial_order=3)
# Associate the sampler with the campaign
my_campaign.set_sampler(my_sampler)
# Will draw all (of the finite set of samples)
my_campaign.draw_samples()
# Encode and execute all runs
my_campaign.populate_runs_dir()
my_campaign.apply_for_each_run_sir(uq.actions.ExecuteLocal(
      "cooling_model.py cooling_in.json"))
# Aggregate decoded output for all runs
my_campaign.collate()
# Post-processing analysis
my_analysis = uq.analysis.PCEAnalysis(sampler=my_sampler, qoi_cols=["te"])
my_campaign.apply_analysis(my_analysis)
# Get Descriptive Statistics
results = my_campaign.get_last_analysis()
stats = results['statistical_moments']["te"]
per = results['percentiles']["te"]
sobols = results['sobols_first']["te"]

In the above, the output of the cooling_model is the temperature, ‘te’ (T at the end of the simulation).

Parameter space definition

The parameter space is defined using a dictionary. Each entry in the dictionary follows the format:

"parameter_name":      {
      "type":      "<datatype>",
      "min":        <value>,
      "max":        <value>,
      "default":    <value>
}

with a defined type, minimum and maximum value and default. If the parameter is not selected to vary in the Sampler (see below) then the default value is used for every run.

App creation

In this example the GenericEncoder and SimpleCSV, both included in the core EasyVVUQ library, were used as the encoder/decoder pair for this application. GenericEncoder performs simple text substitution into a supplied template, using a specified delimiter to identify where parameters should be placed. The template is shown below (is used as the delimiter). The template substitution approach is likely to suit most simple applications but in practice many large applications have more complex requirements, for example the multiple input files or the creation of a directory hierarchy. In such cases, users may write their own encoders by extending the BaseEncoder class.

{
      "T0":"temp_init",
      "kappa":"$kappa",
      "t_env":"$t_env",
      "out_file":"$out_file"
}

As can be inferred from its name, SimpleCSV reads CVS files produced by the cooling model code. Again, many applications output results in different formats, potentially requiring bespoke Decoders.

In this workflow all application runs will be analyzed as individual data points, so we set the collator to AggregateSamples without averaging. This element simply extracts information using the assigned decoder and adds it to a summary dataframe.

The sampler

The user specifies which parameters will vary and their corresponding distributions. In this case the kappa and t_env parameters are varied, both according to a uniform distribution:

vary = {
      "kappa": cp.Uniform(0.025, 0.075),
      "t_env": cp.Uniform(15, 25)
}

Once created, the sampler is associated to the campaign object using the set_sampler() method. Calling the campaign’s draw_samples() method will cause the specified number of samples to be added as runs to the campaign database, awaiting encoding and execution. If no arguments are passed to draw_samples() then all samples will be drawn, unless the sampler is not finite.

Execute runs

Due to the diverse range of execution patterns required by large HPC workflows, EasyVVUQ does not intend to handle matters of execution, leaving this instead to the user’s chosen middleware approach. However, some very basic methods are provided to aid in local execution for testing purposes. The populate_runs_dir() method of my_campaign will create a directory hierarchy containing the encoded input files for every run that has not yet been completed. Finally, in this example, a shell command is executed in each directory to execute the simple test code. In practice this stage would be best handled using, for example, a pilot job manager.

Collation and analysis

Calling my_campaign.collate() at any stage causes the campaign to aggregate decoded simulation output for all runs which have not yet been collated. This collated data is stored in the campaign database. An analysis element can then be applied to the campaign’s collation result. The output of this analysis is dependent on the type of analysis element employed.

In the example application described above, we apply the PCEAnalysis element – the logical counterpart to the PCESampler used earlier to generate the samples. The output of this analysis (‘results’ in the above example script) is a python dict containing the statistical moments (mean, standard deviation etc.) of the Quantity of Interest – in this case, the Temperature, T(‘te’ in the script) – and the associated percentiles of the distribution. The results dict also contains the Sobol indices [17], which measure the sensitivity of the model output (T) to each of the input parameters being varied. Note that, in the script above, this final temperature is referred to as ‘te’. In this case the PCE is being carried out over two inputs, κ and T_env, so the ‘sobols_first’ dict entry contains two values.

Here, the only model output was T, so the analysis element is instructed to operate only on this one output. In general, however, a model can have multiple outputs. In such a case the PCEAnalysis output dict would contain the relevant statistics for each of the outputs, obtainable through using a variable name other than [“te”] as index.

For illustration purposes, the statistical moments (mean, standard deviation and 90% prediction interval) and the first-order Sobol indices for the sensitivity analysis (SA) of the Cooling Coffee Cup model have been plotted in Figure 4. As expected, the mean temperature T is decreasing exponentially towards the environment temperature T_env. For the SA as can be observed (when the first Sobol index is close to 1), T is more affected at the beginning of the simulation by the uncertainty in the rate κ, while it is exclusively affected at the end by the uncertainty in T_env.

Code repository

Name: EasyVVUQ

Persistent identifier: https://github.com/UCL-CCS/EasyVVUQ

Licence: LGPLv3

Date published: 14/12/2018 (date of v0.1 release)