Expert Elicitation Made Easy with ELICIPY, an Online Python Tool

In the world of scientific decision-making, some of the most consequential judgments are made not by machines or direct measurements, but by panels of experts weighing incomplete evidence. Estimating the likelihood of a volcanic eruption, the failure of an engineered barrier, or the spread of a novel pathogen often demands structured expert elicitation—the formal practice of converting expert opinion into calibrated probabilistic estimates. A team of Italian researchers led by Mattia de’ Michieli Vitturi, Andrea Bevilacqua, Alessandro Tadini and Augusto Neri has now unveiled version 2.0 of ELICIPY, an open-source Python tool designed to make this delicate process more rigorous, transparent, and accessible. The update, published in the journal SoftwareX, arrives at a moment when expert judgment is increasingly called upon in high-stakes fields such as volcanic hazard assessment, and it introduces mathematical refinements that could change how large panels of specialists are weighed against one another.

ELICIPY’s core architecture consists of two tightly linked components. The first is a web application, built on the Streamlit framework, through which experts answer calibration questions—so-called seed questions whose true answers are known—and target questions, whose answers are the quantities of interest. The second is an analysis engine that processes the responses, assigns weights to each expert based on their statistical performance, and aggregates the weighted opinions into a single “decision-maker” distribution. Version 2 keeps this familiar structure but substantially upgrades both sides, and the entire tool is now distributed as a standard, pip-installable Python package with three command-line entry points: elicipy-form for building questionnaires, elicipy for running the core analysis, and elicipy-dashboard for exploring the results.

Perhaps the most intellectually significant addition is the new agreement index, a quantitative measure of how closely the experts’ uncertainty intervals overlap. For each pair of experts, the index compares their stated inter-quantile ranges, computed from the lower and upper bounds of each expert’s interval as a normalized overlap ratio that ranges from minus one to one. A value of one means the two intervals coincide perfectly, zero means they are adjacent but non-overlapping, and negative values flag experts whose ranges are separated, indicating genuine conflict. This matters because aggregated uncertainty in a final estimate can arise from two very different situations: experts who genuinely disagree, or experts who individually hedge with very wide ranges. Distinguishing between these cases allows a decision-maker to understand whether the problem is divergent knowledge or simply cautious phrasing. The software reports the mean, standard deviation, and quantiles of pairwise index values under equal weighting and under other weighting schemes, and in related volcanic hazard work the same quantity has been described as a “Conformity Score.” When applied to a test case based on Cotopaxi volcano in Ecuador, the index clearly exposed pairs of experts whose intervals failed to overlap on some target questions while confirming strong consensus on others.

The second headline refinement targets the Classical Model—Cooke’s method—the gold-standard technique for performance-based expert weighting since its introduction in the early 1990s. Under the standard formulation, each expert’s calibration is scored by counting how many realizations of seed questions fall into discrete quantile bins, with the ideal expectation that roughly 5 percent of realizations land below the expert’s 5 percent bound, 45 percent within the 5-to-50 percent range, 45 percent in the 50-to-95 percent range, and 5 percent above the 95 percent bound. The trouble, the authors explain, is that this counting is inherently stepwise: a hair’s-breadth shift in an expert’s estimate, or a slight relocation of a seed realization, can abruptly jump an expert’s weight across a bin boundary. Worse, the discrete counting introduces a granularity error—sometimes called integer bias—when the number of seed questions is small or odd, because it becomes mathematically impossible to realize the theoretically balanced 5-45-45-5 percent distribution.

ELICIPY v2 addresses this with two new formulations offered as advanced options alongside the original method. The Balanced Classical Model handles the awkward edge case in which an expert’s stated value coincides exactly with the true realization: instead of forcing the realization into a single bin, it is split evenly between the two adjacent inter-quantile ranges. The Continuous Classical Model goes further by replacing each seed realization with an “influence interval” whose half-width is defined as one over the number of seed questions, scaled by the ratio of the expert’s 90 percent confidence interval width to 0.90. In other words, the smoothing applied to each realization grows with the expert’s own expressed uncertainty and shrinks as the number of calibration questions increases. By computing the fractional overlap of these intervals with the quantile boundaries, the software produces expert performance as a continuous, smooth function of both judgment and data. The authors draw a conceptual analogy to the transition from Voronoi tessellations to natural-neighbor interpolation: just as Voronoi diagrams are discontinuous with respect to input points, the standard Classical Model jumps at bin boundaries, whereas the new approach yields a smooth response surface. Crucially, the method converges back to the standard Cooke’s Method as the number of seed questions grows, meaning it refines rather than replaces the classical framework—particularly valuable in data-scarce settings where only a handful of seed questions are available.

Beyond the mathematics, the update focuses heavily on usability. A new interactive dashboard, launched with a single command, transforms what was once a static set of output files into an exploratory web interface. Users can browse combined cumulative distribution functions and histograms for any target question, toggle between weighting schemes—Cooke weighting, the ERF scheme, or simple equal weighting—within a single chart, hover over data points for precise values, and zoom into distribution tails. The dashboard automatically filters selectable questions to only those with compatible units and scales, steering users away from meaningless comparisons, and every plot or table can be exported as a PNG image or CSV file for downstream analysis. New visualization options, including violin plots and pie charts of median values, aid side-by-side comparison across groups of questions.

The release also rounds out the elicitation workflow itself. Analysts can now import expert weights from an external file, opening the door to weighting schemes derived from other models or institutional criteria. New output files automatically save percentile values from the 1st to the 99th for each target question, in a separate CSV per weighting scheme, and results are organized into tidy subfolders. Recognizing that the original ELICIPY paper recommended experts draft answers on paper before typing them in, the developers added a utility that reads the questionnaire file and generates fillable PDF forms for both seed and target questions. Online forms can now be password-protected to ensure only authorized panelists respond, and a public webpage tracks repository accesses and clones, giving the team visibility into how the tool is being adopted across the community.

The software’s engineering credentials have been formally certified as well. ELICIPY v2 resides in its own GitHub repository, compliant with the standards defined by the European Open Science Cloud Synergy initiative, and the repository was awarded an SQAaaS Gold Badge following a Quality Assessment and Awarding procedure—a signal of the project’s adherence to reproducibility and software quality benchmarks. The tool requires Python 3.10 or later, is released under the GPL-2.0 license, and is supported through a dedicated email address maintained by the Italian National Institute of Geophysics and Volcanology.

The practical stakes are illustrated by the software’s use in real volcanic risk studies. The updated codebase draws its examples from an expert elicitation supporting hazard and risk assessment for Kolumbo volcano in Greece, and earlier versions of the tool underpinned eruption-probability estimates at Cotopaxi and Guagua Pichincha in Ecuador. In these contexts, structured elicitation bridges the gap between sparse geological data and the probability distributions that emergency planners and civil protection authorities need. By quantifying disagreement among experts and smoothing out the statistical artifacts of small calibration sets, ELICIPY v2 aims to make the bridge sturdier. The development was funded through the Italian “Pianeta Dinamico – Working Earth” program, the Italian Department of Civil Protection, and the Hellenic Survey of Geology and Mineral Exploration, with the authors acknowledging the guidance of elicitation pioneers Willy Aspinall and Roger Cooke.

As uncertainty quantification spreads from volcanology into climate impact studies, pandemic modeling, and engineering risk analysis, tools like ELICIPY are quietly becoming part of the scientific infrastructure. Version 2’s blend of statistical refinement, interactive visualization, and standards-compliant packaging suggests that the craft of asking experts the right questions—and weighting their answers fairly—is maturing into a computational science in its own right.