About UQ Test Functions

This page provides some background information about the what and why of uncertainty quantification (UQ) test functions, their role within UQ analysis methods development, how to usually get them, as well as the motivation behind UQTestFuns.

What are UQ test functions

Consider the following analytical function that computes the maximum annual underflow of a river:

\[\begin{split} \begin{align} \mathcal{M}(\boldsymbol{x}) & = z_v + h - h_d - c_b\\ h & = \left[ \frac{q}{b k_s \left(\frac{z_m - z_v}{l} \right)^{0.5}} \right]^{0.6} \end{align} \end{split}\]

where \(\boldsymbol{x} = \{ q, k_s, z_v, z_m, h_d, c_b, l, b \}\) is the eight-dimensional vector of input variables. The output is given in \([\mathrm{m}]\). A negative value indicates that an overflow (flooding) occurs. This test function is known as the flood model [IL15].

The inputs of a UQ test function is probabilistic, modeled as random variables

The input variables of the function are modeled probabilistically as eight independent random variables with univariate marginals specified in the table below.

No.	Name	Distribution	Parameters	Description
1	\(q\)	Truncated Gumbel (max.)	\([1013, 558, 500, 3000]\)	Maximum annual flow rate \([\mathrm{m^3/s}]\)
2	\(k_s\)	Truncated Normal	\([30, 8, 15, \infty]\)	Strickler coefficient \([\mathrm{m^{(1/3)}/s}]\)
3	\(z_v\)	Triangular	\([49, 51, 50]\)	River downstream level \([\mathrm{m}]\)
4	\(z_m\)	Triangular	\([54, 56, 55]\)	River upstream level \([\mathrm{m}]\)
5	\(h_d\)	Uniform	\([7, 9]\)	Dyke height \([\mathrm{m}]\)
6	\(c_b\)	Triangular	\([55, 56, 55.5]\)	Bank level \([\mathrm{m}]\)
7	\(l\)	Triangular	\([4990, 5010, 5000]\)	Length of the river stretch \([\mathrm{m}]\)
8	\(b\)	Triangular	\([295, 305, 300]\)	River width \([\mathrm{m}]\)

In a calibration (inverse quantification) analysis, the task is to obtain the probabilistic input (posterior) given a function, input prior, and some observed data.

UQ test functions are unique in this way: the input variables are modeled probabilistically as a joint multivariate random variable. In most cases, the specification of the inputs (the distribution, parameters, and dependence structure) are given as part of the problem specification.

Some typical questions in UQ analyses

Given both the function and the input specification, in a typical UQ analysis, we would ask the following questions:

• what is the mean and variance of the output? (uncertainty propagation)

• which input variables drive the uncertainty in the output? (sensitivity analysis)

• what is the probability that an overflow occurs? (reliability analysis / rare event estimation)

Granted, this model is an oversimplification of the real situation and most probably not used to make any real-world decision. But as far as a test function goes, this function exhibits some challenging features for a UQ analysis method. Specifically, the function:

• is multi-dimensional

• contains non-uniform random variables

• involves some interaction terms between the input variables

Before a new UQ method is applied to a real-world problem to answer similar questions as the above, it might be a good idea to check if the method can perform well when applied to the well-known flood model.

Why use UQ test functions

As illustrated above, applied uncertainty quantification in computational science and engineering encompasses many activities, including uncertainty propagation, sensitivity analysis, reliability analysis, optimization, etc. New methods for each of the UQ analyses are continuously being developed. Although such a method is eventually aimed at addressing real-world problems—typically involved a complex expensive-to-evaluate computational model— researchers and developers may prefer to use the so-called test functions during the development phase for validation and benchmarking purposes.

Many UQ test functions have analytical forms, but this is not in any way a requirement

In particular, UQ test functions are widely used in the community for several reasons:

• Test functions are fast to evaluate, at least, faster than the real ones.

• There are many of them available in the literature for various types of analyses.

• Although test functions are taken as black boxes, their features are known; this enables a thorough diagnosis of a UQ method.

• Test functions provide a common ground for comparing the performance of various UQ methods in solving the same class of problems.

Assuming that real models are expensive to evaluate, the cost of analysis is typically measured in the number of function evaluations to achieve the desired accuracy.

Where to get UQ test functions

Online resources to get UQ test functions

Several online resources provide a wide range of test functions relevant to the applied UQ community. For example and by no means an exhaustive list:

• The Virtual Library of Simulation Experiments: Test Functions and Datasets is the definitive repository for UQ test functions. The site provides about a hundred test functions for a wide range of applications, each test function is described in a dedicated page complemented with implementations in MATLAB and R.

• The Benchmark proposals of GdR provide a series of documents that contain test function specifications, but no code implementation whatsoever.

• The Benchmark page of UQWorld provides a selection of test functions in metamodeling, sensitivity analysis, and reliability analysis along with their implementation in MATLAB. Although the implementations themselves are of generic MATLAB, they are geared towards usage in UQLab (a framework for UQ in MATLAB).

• RPrepo—a reliability problems: This repository contains numerous reliability analysis test functions implemented in Python. It is not, however, a stand-alone Python package.

Common to all these online resources are the requirement to either:

• implement the test function oneself following the specification in the programming language of choice, or

• when available, download each of the test functions separately.

Both are neither time-efficient nor convenient.

Alternative sources of UQ test functions: test functions inside an analysis package

Alternatively, in a given programming language, some UQ analysis packages are often shipped with a selection of test functions either for illustration, validation, or benchmarking. Some examples from the applied UQ community in the Python ecosystem are (the data below is as of 2023-06-30):

• SALib includes six test functions mainly for illustrating the package capabilities in the examples [HU17].

• UncertainPy includes 8 test functions (mainly in the context of neuroscience) for illustrating the package capabilities [THE18].

• PyApprox includes 18 test functions, including some non-algebraic functions for benchmarking purposes [Jak21].

• Surrogate Modeling Toolbox (SMT) includes 11 analytical and engineering problems for benchmarking purposes [BHB+19].

• OpenTURNS [BDIP17] has its own separate benchmark package called OTBenchmark that includes 37 test functions [FBCJ21].

Taken together, and still considering some overlaps, all these open-source packages provide quite a lot of test functions already implemented in Python. The problem is that these functions are part of the respective package. To get access to the test functions belonging to a package, the whole analysis package must be installed.

Moreover, test functions from a given package may be implemented in such a way that is tightly coupled with the package itself. To use the test functions belonging to a package, you may need to learn some basic usage and terminologies of the package.

If you are developing a new UQ method and would like to test it against some test functions, going through all of these packages just to get an implementation of (say, algebraic) function sounds like a hassle. You might end up implementing some selection of the test functions yourself and eventually ship them together with your package, just like the other packages.

Why another package

Do we need another package for that?

It seems there is indeed a problem in obtaining UQ test functions implemented in Python. But, a healthy dose of skepticism is, well, healthy; so it’s natural to ask do we need another package for that? UQ test functions are supposed to be available somewhere, at the very least in the literature; their implementations are even already available in some languages. So the question is indeed a valid one.

As exemplified in the list above, some test functions are already implemented in Python and delivered as part of a UQ analysis package. One is even a dedicated benchmark package.

We think “yes”

And yet, we think none of them is satisfactory. Specifically, none of them provides:

• an implementation with minimal dependencies (i.e., NumPy and SciPy) and a common interface of many test functions available in the UQ literature

• a single entry point collecting test functions and their probabilistic input specifications in a single Python package

• an opportunity for an open-source contribution, supporting the implementation of new test functions or posting reference results.

Satisfying all the above requirements is exactly the goal of the UQTestFuns package. In essence, UQTestFuns aims to save the researchers’ and developers’ time from having to reimplement many of the commonly used test functions from the UQ literature themselves. The available functions in UQTestFuns are ready to use for testing and benchmarking purposes.

References

IL15

Bertrand Iooss and Paul Lemaître. A review on global sensitivity analysis methods. In Uncertainty Management in Simulation-Optimization of Complex Systems, pages 101–122. Springer US, 2015. doi:10.1007/978-1-4899-7547-8_5.

HU17

Jon Herman and Will Usher. SALib: an open-source python library for sensitivity analysis. The Journal of Open Source Software, 2(9):97, 2017. doi:10.21105/joss.00097.

THE18

Simen Tennøe, Geir Halnes, and Gaute T. Einevoll. Uncertainpy: a Python toolbox for uncertainty quantification and sensitivity analysis in computational neuroscience. Frontiers in Neuroinformatics, 2018. doi:10.3389/fninf.2018.00049.

Jak21

J. D. Jakeman. PyApprox: enabling efficient model analysis. Technical Report SAND2022-10458, Sandia National Laboratory, Albuquerque, New Mexico, 2021. URL: https://www.osti.gov/servlets/purl/1879614.

BHB+19

Mohamed Amine Bouhlel, John T. Hwang, Nathalie Bartoli, Rémi Lafage, Joseph Morlier, and Joaquim R. R. A. Martins. A Python surrogate modeling framework with derivatives. Advances in Engineering Software, 135:102662, 2019. doi:10.1016/j.advengsoft.2019.03.005.

BDIP17

Michaël Baudin, Anne Dutfoy, Bertrand Iooss, and Anne-Laure Popelin. OpenTURNS: an industrial software for uncertainty quantification in simulation. In Handbook of Uncertainty Quantification, pages 2001–2038. Springer International Publishing, 2017. doi:10.1007/978-3-319-12385-1_64.

FBCJ21

Elias Fekhari, Michaël Baudin, Vincent Chabridon, and Youssef Jebroun. OTBenchmark: an open source Python package for benchmarking and validating uncertainty quantification algorithms. In The 4th ECCOMAS Thematic Conference on Uncertainty Quantification in Computational Sciences and Engineering (UNCECOMP 2021). Eccomas Proceedia, 2021. doi:10.7712/120221.8034.19093.