Expert judgement is routinely required to inform critically important decisions. While expert judgement can be remarkably useful when data are absent, it can be easily influenced by contextual biases which can lead to poor judgements and subsequently poor decisions. Structured elicitation protocols aim to: (1) guard against biases and provide better (aggregated) judgements, and (2) subject expert judgements to the same level of scrutiny as is expected for empirical data. The latter ensures that if judgements are to be used as data, they are subject to the scientific principles of review, critical appraisal, and repeatability. Objectively evaluating the quality of expert data and validating expert judgements are other essential elements. Considerable research suggests that the performance of experts should be evaluated by scoring experts on questions related to the elicitation questions, whose answers are known a priori. Experts who can provide accurate, well-calibrated and informative judgements should receive more weight in a final aggregation of judgements. This is referred to as performance-weighting in the mathematical aggregation of multiple judgements. The weights depend on the chosen measures of performance. We are yet to understand the best methods to aggregate judgements, how well such aggregations perform out of sample, or the costs involved, as well as the benefits of the various approaches. In this paper we propose and explore a new measure of experts’ calibration. A sizeable data set containing predictions for outcomes of geopolitical events is used to investigate the properties of this calibration measure when compared to other, well established measures.

@en

Our main result gives the asymptotic distribution of the determinant of a random correlation matrix sampled in a particular way from the space of d×d correlation matrices. Several spin-off results are proven along the way, and an interesting connection with the law of the determinant of general random matrices is investigated. As different methods for generating random correlation matrices are proposed in the literature, one application of our result is that in can be employed to differentiate between those methods.

@en

Emergency management and long-term planning in coastal areas depend on detailed assessments (meter scale) of flood and erosion risks. Typically, models of the risk chain are fragmented into smaller parts, because the physical processes involved are very complex and consequences can be diverse. We developed a Bayesian network (BN) approach to integrate the separate models. An important contribution is the learning algorithm for the BN. As input data, we used hindcast and synthetic extreme event scenarios, information on land use and vulnerability relationships (e.g., depth-damage curves). As part of the RISC-KIT (Resilience-Increasing Strategies for Coasts toolKIT) project, we successfully tested the approach and algorithm in a range of morphological settings. We also showed that it is possible to include hazards from different origins, such as marine and riverine sources. In this article, we describe the application to the town of Wells-next-the-Sea, Norfolk, UK, which is vulnerable to storm surges. For any storm input scenario, the BN estimated the percentage of affected receptors in different zones of the site by predicting their hazards and damages. As receptor types, we considered people, residential and commercial properties, and a saltmarsh ecosystem. Additionally, the BN displays the outcome of different disaster risk reduction (DRR) measures. Because the model integrates the entire risk chain with DRR measures and predicts in real-time, it is useful for decision support in risk management of coastal areas.

@en

In decision and risk analysis problems, modelling uncertainty probabilistically provides key insights and information for decision makers. A common challenge is that uncertainties are typically not isolated but interlinked which introduces complex (and often unexpected) effects on the model output. Therefore, dependence needs to be taken into account and modelled appropriately if simplifying assumptions, such as independence, are not sensible. Similar to the case of univariate uncertainty, which is described elsewhere in this book, relevant historical data to quantify a (dependence) model are often lacking or too costly to obtain. This may be true even when data on a model’s univariate quantities, such as marginal probabilities, are available. Then, specifying dependence between the uncertain variables through expert judgement is the only sensible option. A structured and formal process to the elicitation is essential for ensuring methodological robustness. This chapter addresses the main elements of structured expert judgement processes for dependence elicitation. We introduce the processes’ common elements, typically used for eliciting univariate quantities, and present the differences that need to be considered at each of the process’ steps for multivariate uncertainty. Further, we review findings from the behavioural judgement and decision making literature on potential cognitive fallacies that can occur when assessing dependence as mitigating biases is a main objective of formal expert judgement processes. Given a practical focus, we reflect on case studies in addition to theoretical findings. Thus, this chapter serves as guidance for facilitators and analysts using expert judgement.@en

Probabilistic thinking can often be unintuitive. This is the case even for simple problems, let alone the more complex ones arising in climate modelling, where disparate information sources need to be combined. The physical models, the natural variability of systems, the measurement errors and their dependence upon the observational period length should be modelled together in order to understand the intricacies of the underlying processes. We use Bayesian networks (BNs) to connect all the above-mentioned pieces in a climate trend uncertainty quantification framework. Inference in such models allows us to observe some seemingly nonsensical outcomes. We argue that they must be pondered rather than discarded until we understand how they arise. We would like to stress that the main focus of this paper is the use of BNs in complex probabilistic settings rather than the application itself.@en

Many applications in decision making under uncertainty and probabilistic risk assessment require the assessment of multiple, dependent uncertain quantities, so that in addition to marginal distributions, interdependence needs to be modelled in order to properly understand the overall risk. Nevertheless, relevant historical data on dependence information are often not available or simply too costly to obtain. In this case, the only sensible option is to elicit this uncertainty through the use of expert judgements. In expert judgement studies, a structured approach to eliciting variables of interest is desirable so that their assessment is methodologically robust. One of the key decisions during the elicitation process is the form in which the uncertainties are elicited. This choice is subject to various, potentially conflicting, desiderata related to e.g. modelling convenience, coherence between elicitation parameters and the model, combining judgements, and the assessment burden for the experts. While extensive and systematic guidance to address these considerations exists for single variable uncertainty elicitation, for higher dimensions very little such guidance is available. Therefore, this paper offers a systematic review of the current literature on eliciting dependence. The literature on the elicitation of dependence parameters such as correlations is presented alongside commonly used dependence models and experience from case studies. From this, guidance about the strategy for dependence assessment is given and gaps in the existing research are identified to determine future directions for structured methods to elicit dependence.@en

The length of a disruption in a railway network is highly uncertain which complicates the incident management of traffic controllers. This paper proposes a probabilistic model based on historical data to provide the prediction of the disruption length to the Dutch Operational Control Centre Rail (OCCR). A good prediction of disruption length is believed to help the OCCR to implement an appropriate response that minimizes the impact of the disruption for the railway users. The model that is proposed in this paper is a Non-Parametric Bayesian Network (NPBN) which represents the joint distribution between variables that describe the nature of the disruption. To obtain the prediction of the disruption length, this joint distribution is conditionalized on the particular values of variables in the model that are describing the situation at hand. The NPBN allows rapid conditionalization/inference which is attractive for the real-time decision making process of the OCCR. This paper presents the first attempt to model disruption length with NPBNs. A case study concerning a specific type of railway disruption, namely malfunctioning train detection, is considered as an example of the application of the method.@en