The study examines a stage often omitted from legal AI benchmarks. Existing legal-reasoning tests generally begin with facts, legal texts, rules or questions already supplied, then measure tasks such as issue spotting, rule recall, rule application, interpretation or conclusion generation. Real legal consultations can start differently: clients may provide partial, emotional or non-linear accounts, overlook details they consider unimportant, or disclose important information only after structured follow-up questions. The researchers call the process of eliciting, clarifying, stabilizing and narrating those details “fact construction.” They argue that this pre-reasoning work should be evaluated separately because a fluent legal answer can still rest on an incomplete, distorted or internally inconsistent factual premise.

To make fact construction observable, the team designed four distinct roles. A fact-constrained “Litigant Twin” acted as a proxy client, holding the original record but releasing information only when asked and indicating when a requested detail was absent, unclear or uncertain. It was not intended to reproduce a person’s personality, memory, preferences or strategic behavior. A “LegalAider” conducted structured, non-redundant questions about actors, timelines, events, transactions, injuries and procedural developments. A “NarrativeAgent” converted the resulting dialogue into a coherent, neutral factual account without making legal conclusions. An “Evaluator” then compared that account with the original human-authored record, using fixed criteria for accuracy, completeness and consistency while flagging omissions, contradictions, unsupported additions and uncertainty.

The records were written in Chinese as electronic forms for legal clinic consultations conducted during the 2021–2023 academic years, with consultation dates ranging from September 2021 to May 2024. They were not verbatim transcripts of lawyer-client conversations or adjudicated findings of fact, but institutional records prepared during clinical legal service. For the experiment, the fact-constrained client received only the student-prepared case-facts field and the consultation seeker’s questions. Proposed legal responses, advice and rule references were deliberately excluded. The sample included consumer disputes, family matters, debt and labor disputes, traffic accidents, criminal matters, property disputes and tort-liability claims. Because records were selected for sufficient factual completeness to permit meaningful comparison, this was a criterion-based dataset rather than a random sample of all consultations.

De-identification was both a privacy requirement and a source of experimental difficulty. Personal names, telephone numbers, addresses, institutional identifiers and court case numbers were removed or replaced with neutral labels such as “Party A,” “Party B” or “Company X.” The experiment used a three-by-three evaluator-generator matrix: each generator condition used one model for the Litigant Twin, LegalAider and NarrativeAgent, while the evaluator model was varied independently. After missing or invalid evaluator outputs were excluded, 889 valid scores remained. Repeated-measures analyses used 89 cases with valid scores across all nine evaluator-generator combinations. The design measures the complete workflow rather than isolating the contribution of any single agent, because changes in disclosure, questioning or synthesis could all affect the final narrative.

Mean factual-alignment scores ranged from 0.70 to 0.88, with standard deviations between 0.07 and 0.15. Claude-3.5-Haiku was the more lenient evaluator in all three generator conditions, while the other evaluators returned lower averages. Friedman tests on case-aggregated scores indicated a generator effect and a stronger evaluator effect, both with p-values below 0.001. The authors said this stronger evaluator effect showed that score distributions were substantially shaped by differences in evaluator calibration. Spearman rank correlations between evaluators were positive but moderate, ranging from 0.54 to 0.77, showing that the models shared some ranking tendencies without ordering narratives consistently. A higher score therefore did not necessarily mean that a narrative was objectively superior: evaluators could differ in leniency, in their treatment of omissions and in how they defined factual alignment.

Inspection of low-scoring and unstable cases identified four recurring failure modes. First, anonymized placeholders sometimes broke the mapping between actors and their legal or factual roles. Questions could be directed toward the wrong person, and the synthesis agent could then produce a sparse or confused account. Second, third-party accounts placed information between the clinic client and the relevant events. The constrained proxy appropriately responded that certain details were unavailable, but this could preserve uncertainty so strongly that the resulting narrative was less complete than the original institutional record. Third, multi-episode matters were sometimes collapsed into a single simplified story when the questioning agent failed to establish separate incidents, dates, medical examinations, reports and procedural developments. Fourth, monetary amounts, compensation calculations, commissions, asset values and repair or medical costs were easily omitted or compressed, while evaluators differed markedly in how heavily they penalized those losses.

Qualitative examples showed why a numerical score cannot identify the cause of a mismatch on its own. A consumer consultation receiving a score of 0.92 was described as almost fully aligned with the original facts, whereas a mental-health-related case receiving 0.30 showed substantial divergence from the original factual structure. Two cases assigned the same mid-range score of 0.62 represented different situations: one preserved the core account but omitted some details, while the other contained omissions and weak narrative structure. A coherent surface narrative can conceal missing chronology, confused actors or omitted calculations. The study consequently separated factual adherence from legal relevance, issue selection, doctrinal classification, advice quality and outcome prediction. It did not claim that every preserved detail was necessarily legally important, only that the first evaluation layer concerns whether available facts remain aligned with the source account.

The findings do not establish that multi-agent role separation outperforms a simpler system because the study included no direct single-agent summarization baseline. It also used one composite score informed by accuracy, completeness and consistency rather than independently validated measures for each dimension. The dataset came from one Taiwanese clinical setting, and the sensitive consultation records cannot be released in full, limiting direct replication. Most importantly, model-driven scores were not systematically calibrated against human expert judgments. The researchers therefore described the framework as a practice-grounded diagnostic and simulation tool for making elicitation, narrative construction, evaluator disagreement and failure modes visible. They also note that the Litigant Twin, LegalAider and NarrativeAgent used the same model within each generator condition, so causal responsibility for a particular error cannot be assigned to one component.

Within these limits, the work frames large language models as tools for supervised factual support rather than autonomous legal advisers. The authors see potential value in early-stage intake, factual organization and diagnostic review, where a provisional narrative could help professionals identify missing questions, contradictions and uncertainty. Human oversight would still be required to trace the narrative to the source record, inspect the dialogue and verify disputed details before legal assessment, counseling, drafting or strategic decisions. The study’s broader implication is that legal AI reliability depends on more than model capability: structured questioning, stable role mapping, explicit separation of episodes, careful retention of numerical details, transparent evaluation and reviewable institutional records all shape the factual foundation available to downstream legal analysis.

Subject of Research: Evaluation of multi-agent fact construction in AI-assisted legal consultations using 100 de-identified clinical records

Article Title: Fact construction as a missing layer in legal AI: a practice-grounded framework for evaluating multi-agent legal consultation

Article References: Shao, H.-L., Yu, Y.-C., & Huang, S.-C. (2026). Fact construction as a missing layer in legal AI: a practice-grounded framework for evaluating multi-agent legal consultation. Frontiers in Artificial Intelligence, 9, Article 1896166. https://doi.org/10.3389/frai.2026.1896166

Image Credits: AI Generated

DOI: 10.3389/frai.2026.1896166

Keywords: legal AI, fact construction, multi-agent systems, large language models, legal consultations, factual alignment, LLM evaluation, human oversight, clinical legal records, Fact, construction, missing