Time series models are common in science, where they are used to describe the dynamics of system behaviours. In many cases, these models are non-linear and impossible to solve analytically, so that the forward problem (predicting the model output for a given set of parameters) is computationally hard. For such models, there is no single method which can reliably solve the inverse problem of estimating parameter values from a noisy time trace. Much like there is a variety of forward models, there is a diversity of approaches for parameter inference. Further, it is often unclear which approach to apply when, meaning that researchers are required to implement a range of methods before successfully fitting their model to data.
PINTS is a software framework that allows users to easily trial and apply different inference methods to their problem. The inference methods supplied by PINTS fall into two broad categories: optimisers, which attempt to find a single best parameter vector, and samplers, which aim to estimate a probability distribution over parameter values that are compatible with observed results. Users are expected to already have a forward model (for example, a simulation) at their disposal, which they make available to PINTS by writing a simple Python wrapper. They then define a Problem (a forward model plus a data set), on which either an ErrorMeasure (for optimisation) or a LogPDF (for optimisation and sampling) is defined. Currently available optimisers include CMA-ES , XNES , SNES , and Particle Swarm Optimisation (PSO) . Sampling methods include Random Walk Metropolis Markov chain Monte Carlo (MCMC) [13, 15], adaptive covariance MCMC , Population MCMC , Differential Evolution , DREAM , emcee , Hamiltonian MCMC , and MALA . In addition, ellipsoidal  and rejection nested samplers  are provided. Convenience plotting methods are provided to quickly visualise the results, as well as diagnostic tools to inspect the validity of the results. An example of an optimisation problem and its solution using PINTS is shown in Figure 1.
PINTS was developed as a community effort by researchers in electrochemistry, cardiac electrophysiology, and statistics, to compare different methods for solving inverse problems in a common framework. It features a clean and transparent object-orientated API that is designed to easily accommodate new error measures, log-likelihoods, optimisers and samplers, allowing users to utilise pre-built components as much as possible, while adding their own code for problem-specific areas. The PINTS team aims for full test coverage, and includes unit testing and extended statistical tests to verify the correct operation of all methods.
Early research using PINTS in electrochemistry has included fitting a differential-algebraic equation (DAE) model of reduction-oxidation to voltammetry measurements of a Polyoxometalates molecule , and the design and application of a custom hierarchical statistical model for repeat voltammetry experiments of a Ferricynide process .
Optimisation algorithms are implemented in many different software packages, (see, for example, the Python scipy.optimize module), but are often biased towards gradient-based methods, which can perform poorly for many ordinary and partial differential equations used in time series modelling. PINTS therefore focuses on derivative-free optimisers, although we plan to add gradient-based methods for comparison. In contrast with more general-purpose optimisation software, PINTS contains a number of error measures specifically suited to time series models, and adds the ability to use any PINTS log-likelihood class as an error measure in order to perform maximum likelihood estimation (MLE).
Dakota  is a widely regarded package for parameter fitting and uncertainty quantification and is most similar to PINTS in that it offers a generic interface to call an (assumed expensive) model, as well as a wide variety of optimisers and samplers. In contrast to the PINTS Python API, Dakota uses either a C++ or file input/output process for communication between user models and the library, and does not provide options for specifying either the error measures or the log-likelihoods of the inverse problem. However, Dakota has some features not yet available in PINTS, such as the option to train a surrogate model, useful for very expensive model evaluations.
Other software packages that enable parameter inference and sampling for ODE models include BioBayes , ABC-SysBio , SYSBIONS , PyMC3 , and Stan . These packages use either a common model description format (for example, SBML) or their own language (for example, Stan’s probabilistic programming language) to specify the model, presenting additional learning hurdles for a user and often restricting the class of models which can be fit. By contrast, PINTS aims to be as general as possible to support a wider variety of models (for example, PDEs). PyMC3  does provide a similar generic model interface to PINTS but, as with the other packages, specialises in one sampling method, whereas PINTS aims to support a wide variety of methods with the assumption that no one sampling method is suitable for all models of interest. BCM  offers both a generic interface (via C++) and a wide variety of samplers, but does not supply any likelihood functions and is unfortunately largely undocumented.
PINTS is designed around two core ideas: 1. PINTS should work with a wide range of time series models, and make no demands on how they are implemented other than a minimal input/output interface. 2. It is assumed that model evaluation (simulation) is the most costly step in any optimisation or sampling routine.
The decision to use Python fits both these criteria: Python interfaces well with C and C++, which are typically used for high-performance simulation, and any performance hit of using the high-level, easy to read and write language Python is overshadowed by simulation time.
All optimisers operate on a callable ErrorMeasure object that describes a function to minimise or on a callable LogPDF object that describes a probability density function (PDF) to maximise. Similarly, all samplers start from a callable LogPDF, so that the same probability function can be used with both optimisers and samplers. The natural logarithm of the PDF is used for computational efficiency and accuracy, and we allow the probability density to be unnormalised (i.e. its integral does not have to sum to 1). Figure 2 shows how a user-defined model can be wrapped in a PINTS ForwardModel and combined with time points and measured values to create a Problem from which several standard ErrorMeasures and LogPDFs can be created. For inference in a Bayesian context, a LogPosterior class and several LogPrior distributions are provided. If a given LogPDF or ErrorMeasure cannot be constructed from PINTS classes, users can also define their own classes.
Most PINTS samplers and optimisers are implemented using a so-called ask-and-tell interface, inspired by the Python implementation of CMA-ES  (https://github.com/CMA-ES/pycma). In this framework, the details of solving the forward problem are partitioned away from the rest of the sampling or optimising algorithm. For each iteration the following steps are undertaken (Figure 3): first, the user calls ask() to obtain one or more parameter values from their chosen method — these values are typically vectors generated stochastically conditional on an internal system state; second, the user solves the forward model and generates a score for each parameter vector, for example, an error measure or (unnormalised) posterior probability; third, the user calls tell() to pass the score back to the method, which can then update its internal state and finish the iteration. For example, in many MCMC methods each ask() call returns a single proposed sample to be evaluated by the user, and the following tell() then either accepts or rejects this point based on its probability. For optimisers such as CMA-ES, ask() returns a set of points in parameter space, and the scores passed in via tell() are then used to estimate the local gradient, which is used to move towards the estimated optimum. This framework has a number of advantages: since optimisation or sampling can take hours or even days, this allows programs using PINTS to provide regular user feedback and logging (which is not possible when the routine is implemented as a single monolithic function call); allows users with access to CPU clusters or GPU machines to implement their own parallelised evaluation of ErrorMeasures and LogPDFs; lets users implement their own strategies (for example, by dynamically changing hyperparameters) and/or stopping criteria; and, finally, by delineating the sampling or optimisation algorithm’s steps from the methods used to solve the forward problem, encourages development of transparent and modular code.
For more casual users (whom we expect will be the majority), PINTS includes ‘controller’ classes (e.g. OptimisationController MCMCController) that provide a higher level interface (see e.g. Figure 1). When creating a controller, the user specifies the name of the lower-level method to use (e.g. CMAES, or AdaptiveCovarianceMCMC) and passes in an ErrorMeasure or LogPDF. The controller then instantiates and manages this method, as well as providing user-configurable stopping criteria (e.g. maximum number of iterations) and logging to screen and/or the filesystem. An important difference with the lower-level ask-and-tell interface, is that controllers handle function evaluation, and can distribute evaluations over multiple CPU cores using Python multiprocessing. This provides users with out-of-the-box parallelisation, while users looking to implement custom parallelisation can fall back on the ask-and-tell interface.
PINTS has three levels of testing: unit testing, functional testing, and comparative testing. Unit tests are used to test the functionality of simple (deterministic) methods, and to check that complex (pseudo-random) methods run without raising exceptions. All the unit tests are available to be run by a user to ensure the software is working correctly. Continuous integration is carried out using Travis CI (Ubuntu Trusty distribution with Python versions 2.7, 3.4, 3.5 and 3.6, and OS/X with Python version 2.7) and AppVeyor (Windows Server 2011 R2 with Python versions 2.7, 3.4, 3.5 and 3.6). PINTS uses Flake8 linter tests to ensure that contributed code conforms to best practices, and code coverage tests to ensure that all code is sufficiently tested and documented. Functional testing of the methods is performed separately, and is used to test the method’s behaviour from different (pseudo-random) initial conditions. Analysis of functional checking is done both visually and statistically (for example, if recent results deviate significantly from previous results this indicates the possible introduction of a bug). In our current solution, functional checking is run automatically but the results are interpreted by the PINTS developers. Work is ongoing to develop an automated analysis system so that these functional tests can be included in the continuous integration pipeline. Finally, in comparative testing a number of problems are set up and solved with different methods, in order to compare the solutions they return and evaluate their performance. Performance testing is partially automated, but tests are initiated and analysed by the developer.
PINTS uses no functions specific to any operating system (OS) and so can run on any OS that provides Python. Optional parallelisation is provided that uses the Python multiprocessing module, which works best on UNIX-based systems (for example, Linux and OS/X), but runs on Windows with slightly reduced performance.
PINTS requires Python 2.7 or higher, or Python 3.4 or higher.
PINTS has a minimal disk space footprint (approximately 2MB) and can be run on single-processor devices or headless on multi-processor machines (for example, via ssh).
PINTS uses the NumPy (version 1.8 or higher) and SciPy (version 0.14 or higher, ) libraries extensively. The default optimisation method is CMA-ES, for which the cma package (version 2 or higher) is used. The remaining optimisation and sampling methods require no further dependencies. Finally, Matplotlib (version 1.5 or higher ) is required for installation, but it is possible to use PINTS without Matplotlib or with a different plotting library.
Michael Clerx, Sanmitra Ghosh, Ben Lambert, Chon Lok Lei, Martin Robinson.
Name: GitHub (release v0.2.2)
Persistent identifier: https://github.com/pints-team/pints/releases/tag/v0.2.2
Licence: BSD 3-clause
Publisher: Pints team
Version published: 0.2.2
Date published: 30/04/2019
Name: GitHub (develop)
Persistent identifier: https://github.com/pints-team/pints
Licence: BSD 3-clause
Date published: 16/05/17
Detailed documentation is provided on using PINTS with user-supplied models (see, for example, the writing-a-model example on the GitHub repository). While PINTS was designed primarily with biological and electrochemical problems in mind, there is nothing to prohibit its use on time series models from other fields. Similarly, while the implemented ErrorMeasure and LogPDF classes were chosen to work well with time series problems, PINTS can be used outside this setting (in fact, utility functions fmin and curve_fit are provided specifically for this purpose). Users are also free to create their own ErrorMeasure or LogPDF classes that do not rely on PINTS ForwardModel or Problem classes, which allows PINTS to be used beyond its intended scope of time-series problems. A link to the full API documentation can be found on the GitHub repository, which also contains a list of examples for all PINTS’ main features (https://github.com/pints-team/pints/tree/master/examples). We have found that these examples, rather than our API documentation, serve to kick-start any new project based on PINTS.
We welcome questions, suggestions, bug reports, and user contributions via the GitHub repository, which acts as a central communication platform for PINTS. A detailed guide on contributing to PINTS is also available there. A Research Software Engineering group has recently been established at Oxford university, through which we hope to provide long-term support. In addition, we are happy to respond to questions via e-mail or other means of communication.
The authors have no competing interests to declare.
Michael Clerx and Martin Robinson authors contributed equally to this manuscript.
Adam, B, Bauman, L, Bohnhoff, W, Dalbey, K, Ebeida, M, et al. 2015 ‘Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: Version 6.0 user manual’, Technical report, Tech. rep., Sandia NationalLaboratories. DOI: https://doi.org/10.2172/1177048
Carpenter, B, Gelman, A, Hoffman, M D, Lee, D, Goodrich, B, Betancourt, M, Brubaker, M, Guo, J, Li, P and Riddell, A 2017 ‘Stan: A probabilistic programming language’. Journal of Statistical Software, 76(1). DOI: https://doi.org/10.18637/jss.v076.i01
Foreman-Mackey, D, Hogg, D W, Lang, D and Goodman, J 2013 ‘emcee: the MCMC hammer’. Publications of the Astronomical Society of the Pacific, 125(925): 306. DOI: https://doi.org/10.1086/670067
Girolami, M and Calderhead, B 2011 ‘Riemann manifold Langevin and Hamiltonian Monte Carlo methods’. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(2): 123–214. DOI: https://doi.org/10.1111/j.1467-9868.2010.00765.x
Glasmachers, T, Schaul, T, Yi, S, Wierstra, D and Schmidhuber, J 2010 ‘Exponential natural evolution strategies’. In: ‘Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation’, ACM, 393–400. DOI: https://doi.org/10.1145/1830483.1830557
Hansen, N, Müller, S D and Koumoutsakos, P 2003 ‘Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)’. Evolutionary Computation, 11(1): 1–18. DOI: https://doi.org/10.1162/106365603321828970
Hunter, J D 2007 ‘Matplotlib: A 2D graphics environment’. Computing in Science & Engineering, 9(3): 90–95. DOI: https://doi.org/10.1109/MCSE.2007.55
Jasra, A, Stephens, D A and Holmes, C C 2007 ‘On population-based simulation for static inference’. Statistics and Computing, 17(3): 263–279. DOI: https://doi.org/10.1007/s11222-007-9028-9
Johnson, R, Kirk, P and Stumpf, M P 2014 ‘SYSBIONS: nested sampling for systems biology’. Bioinformatics, 31(4): 604–605. DOI: https://doi.org/10.1093/bioinformatics/btu675
Johnstone, R H, Chang, E T, Bardenet, R, De Boer, T P, Gavaghan, D J, Pathmanathan, P, Clayton, R H and Mirams, G R 2016. ‘Uncertainty and variability in models of the cardiac action potential: Can we build trustworthy models?’ Journal of Molecular and Cellular Cardiology, 96: 49–62. DOI: https://doi.org/10.1113/JP271671
Jones, E, Oliphant, T and Peterson, P 2014 ‘Scipy: open source scientific tools for Python’. http://www.scipy.org. [Online; accessed 13-Aug-2018].
Kennedy, J 2011 Particle swarm optimization. In: ‘Encyclopedia of Machine Learning’, Springer, 760–766. DOI: https://doi.org/10.1007/978-0-387-30164-8_630
Liepe, J, Barnes, C, Cule, E, Erguler, K, Kirk, P, Toni, T and Stumpf, M P 2010 ‘ABC-SysBio-approximate Bayesian computation in Python with GPU support’. Bioinformatics, 26(14): 1797–1799. DOI: https://doi.org/10.1093/bioinformatics/btq278
Metropolis, N, Rosenbluth, A W, Rosenbluth, M N, Teller, A H and Teller, E 1953 ‘Equation of state calculations by fast computing machines’. The Journal of Chemical Physics, 21(6): 1087–1092. DOI: https://doi.org/10.1063/1.1699114
Mukherjee, P, Parkinson, D and Liddle, A R 2006 ‘A nested sampling algorithm for cosmological model selection’. The Astrophysical Journal Letters, 638(2): L51. URL: https://arxiv.org/pdf/astro-ph/0508461. DOI: https://doi.org/10.1086/501068
Neal, R M, et al. 2011 ‘MCMC using Hamiltonian dynamics’. Handbook of markov chain monte carlo, 2(11): 2. DOI: https://doi.org/10.1201/b10905-6
Robinson, M, Bond, A, Simonov, A, Zhang, J and Gavaghan, D 2018 ‘Separating the effects of experimental noise from inherent system variability in voltammetry: The [Fe(CN)6]3/4 process’. ChemRxiv. DOI: https://doi.org/10.26434/chemrxiv.7149281.v1
Robinson, M, Ounnunkad, K, Zhang, J, Gavaghan, D and Bond, A 2018 ‘Integration of heuristic and automated parametrization of three unresolved twoelectron surfaceconfined polyoxometalate reduction processes by AC voltammetry’. ChemElectroChem. DOI: https://doi.org/10.1002/celc.201800950
Salvatier, J, Wiecki, T V and Fonnesbeck, C 2016 ‘Probabilistic programming in Python using PyMC3’. PeerJ Computer Science, 2: e55. DOI: https://doi.org/10.7717/peerj-cs.55
Schaul, T, Glasmachers, T and Schmidhuber, J 2011 ‘High dimensions and heavy tails for natural evolution strategies’. In: ‘Proceedings of the 13th annual conference on Genetic and evolutionary computation’, ACM, 845–852. DOI: https://doi.org/10.1145/2001576.2001692
Skilling, J, et al. 2006 ‘Nested sampling for general Bayesian computation’. Bayesian Analysis, 1(4): 833–859. DOI: https://doi.org/10.1214/06-BA127
Ter Braak, C J 2006 ‘A Markov Chain Monte Carlo version of the genetic algorithm Differential Evolution: easy Bayesian computing for real parameter spaces’. Statistics and Computing, 16(3): 239–249. DOI: https://doi.org/10.1007/s11222-006-8769-1
Thijssen, B, Dijkstra, T M, Heskes, T and Wessels, L F 2016 ‘BCM: toolkit for Bayesian analysis of computational models using samplers’. BMC Systems Biology, 10(1): 100. DOI: https://doi.org/10.1186/s12918-016-0339-3
Vrugt, J A, Ter Braak, C, Diks, C, Robinson, B A, Hyman, J M and Higdon, D 2009 ‘Accelerating Markov chain Monte Carlo simulation by differential evolution with self-adaptive randomized subspace sampling’, International Journal of Nonlinear Sciences and Numerical Simulation, 10(3): 273–290. DOI: https://doi.org/10.1515/IJNSNS.2009.10.3.273
Vyshemirsky, V and Girolami, M 2008 ‘BioBayes: a software package for Bayesian inference in systems biology’. Bioinformatics, 24(17): 1933–1934. DOI: https://doi.org/10.1093/bioinformatics/btn338