AI Delphi Panels Converge Reliably but Can Miss Entire Research Fields

A panel of sixty large language model agents, role-playing as experts from materials engineering to circular business models, has passed one of the toughest tests yet applied to synthetic foresight: when its forecasts were checked against what actually happened in the research literature, the panel’s rankings tracked observed publication growth — but only for the parts of the field its evidence base could reach. The study, published in the European Journal of Futures Research, deployed the panel over a knowledge graph of 25,823 circular economy publications and systematically varied what the artificial panellists were allowed to read. The result is both a proof of concept and a warning: a synthetic panel can converge beautifully on an incomplete picture of the world, and none of its internal quality metrics will tell you so.

The experiment addresses a question that has hung over LLM-based Delphi methods since their first proof of concept. Bertolotti and Mari had previously shown that a language model can sustain autonomous multi-round deliberation, playing both facilitator and respondents, but their design used a single model family, had no external knowledge grounding, and could not be validated against any observable outcome because its forecast domain — the future of generative AI — was purely prospective. The authors of the new study took the three extensions those researchers identified as priorities and implemented all of them: a multi-model panel, systematic ablation of retrieval strategies, and validation against a historical record that actually exists.

The panel itself was assembled from six LLM families — Meta’s Llama-3.1-70B, Google’s Gemma-3-27B, Mistral-Large, Anthropic’s Claude Haiku 4.5, Alibaba’s Qwen3-30B-A3B, and Microsoft’s Phi-4 — each instantiated as ten disciplinary personas spanning sustainability science, industrial ecology, construction waste, plastics chemistry, life cycle assessment, and circular business models. The agents ran three classic Delphi rounds: independent elicitation, anonymised feedback through a facilitator summary, and a convergence round. Each panellist produced pessimistic, median, and optimistic growth scores for each of eleven research subdomains, plus three to five emerging terms per cluster.

Convergence was emphatic. Across the valid retrieval conditions, the coefficient of variation among panellists fell by between 81 and 91 percent over three rounds, comfortably exceeding the 60 percent reduction that serves as the standard Delphi quality benchmark. Under the strongest graph-derived condition, mean dispersion dropped from 0.159 in the first round to 0.026 in the third. The dynamic replicated the familiar Delphi pattern: substantial revision after the first shared feedback, then stabilisation. On the surface, the synthetic panel behaved exactly like a well-run human expert panel.

But convergence turned out to measure agreement, not accuracy. The circular economy field is a textbook case of what the authors, borrowing from prior work, call awareness without synthesis: subdomains that cite each other yet use almost entirely non-overlapping vocabularies. Three clusters in the knowledge graph — advanced plastics and biocomposites, recycled construction materials, and industrial waste recovery — showed near-zero citation connectivity to the rest of the field. Under all six standard retrieval-augmented configurations, which ranged from no grounding at all to full-text literature, EU regulatory documents, and pre-extracted weak signals, the panel’s recall for these three isolated subdomains was exactly zero. Across three rounds of deliberation, not one of the sixty panellists surfaced a single ground-truth term for the mechanical properties of biocomposites, the compressive strength of recycled aggregates, or material recovery yields in waste processing.

The failure was not a quirk of language model generation. Independent retrieval tests, run without any LLM in the loop, showed that BM25 keyword search returned zero precision for cross-cluster queries targeting the isolated clusters in 27 of 30 cases, and swapping in ColBERT v2 late-interaction retrieval left 27 of those 30 zeros intact. The problem sits at the vocabulary level, not the architecture level: a retriever cannot find documents whose distinctive terms never co-occur with the query vocabulary anywhere in the corpus. No amount of deliberation can rescue knowledge that was never retrieved, because in a synthetic panel every participant draws from the same context. Deliberation reduces dispersion over what is already on the table; it cannot add to the table.

The remedy came from the structure of the knowledge graph itself. Two topology-aware strategies — bridge-term injection, which feeds panellists distinctive vocabulary shared between isolated and connected clusters via class-based TF-IDF, and bridge-author traversal, which identifies researchers who publish in both worlds and injects their bilingual lexicons — recovered non-zero coverage for the isolated subdomains. The bridge-term condition reached all three isolated clusters, lifting recall for construction materials from zero to 0.75, while the bridge-author condition recovered the plastics cluster. Both strategies require no external ontologies and no manual curation; they extract everything from citation networks and authorship records already present in the graph.

Ranking accuracy told a more cautious story. Panel-derived rankings correlated with observed 2021 to 2024 OpenAlex publication growth at Spearman coefficients ranging from 0.06 to 0.54 across the seven valid conditions, with abstract-only retrieval performing best descriptively. But with only eleven clusters, no condition reached conventional statistical significance, and all bootstrapped confidence intervals spanned zero. The authors are explicit that these correlations are descriptive comparisons of how different evidence bases shift the panel’s ordinal output, not inferential tests of forecast accuracy. The graph-derived configurations traded some ranking precision for coverage, and no single configuration dominated across all clusters — a Pareto trade-off the authors present as the substantive finding of the ablation.

From these results the researchers distil a four-step, audit-first protocol for practitioners. First, run a structural audit of the target field’s citation connectivity before designing the panel, since isolated subdomains predict coverage failure better than any downstream choice. Second, report coverage alongside convergence, in the same way information retrieval reports precision and recall together. Third, choose the retrieval strategy by task objective: completeness-critical work such as gap identification favours the graph-derived conditions, while ordering-critical tasks such as roadmapping may favour text-based retrieval, with absent subdomains disclosed as a known limitation. Fourth, treat bridge vocabularies as standing infrastructure, stored as cluster properties and refreshed as the corpus grows.

The authors are careful about the boundaries of their claims. The analysis rests on eleven clusters, which caps the statistical reach of both the ranking test and the isolation diagnostic, whose threshold was calibrated on only three cases. The ground truth consists of three to four terms per cluster selected by a single domain expert, who may share the vocabulary blind spots of the dominant discourse. The study covers one field and one knowledge graph, validation is retrospective rather than prospective, and no human panel was run on the same evidence base to separate the contribution of retrieval architecture from that of panellist type. The prospective 2030 to 2040 exercise the team ran as a demonstration is presented as an informed projection, not a validated forecast. What survives those caveats is the paper’s central methodological point: when a synthetic panel converges, what it is converging about must be audited independently, because agreement among artificial panellists — however impressive the numbers — is not, by itself, evidence that they have seen the whole field.

Subject of Research: Empirical validation of synthetic LLM-based Delphi panels for participatory foresight using a circular economy knowledge graph

Article Title: When synthetic Delphi panels work: a graph-native methodology for participatory foresight under structural complexity

Article References: Bossler, A., Fullana, A., & Bas, E. (2026). When synthetic Delphi panels work: a graph-native methodology for participatory foresight under structural complexity. European Journal of Futures Research, 14(1), Article 38. https://doi.org/10.1186/s40309-026-00302-x

Image Credits: AI Generated

DOI: 10.1186/s40309-026-00302-x

Keywords: synthetic Delphi, large language models, foresight methodology, retrieval-augmented generation, circular economy, knowledge graphs, panel convergence, subdomain coverage, bibliometrics, participatory foresight, EU policy, validation