Elevator systems serve billions of passenger journeys across the globe every day, functioning as indispensable yet highly complex cyber-physical infrastructures that combine intricate mechanical assemblies, electrical circuits, and computerized controllers. Because these systems are subject to continuous mechanical wear, environmental fluctuations, and component degradation, identifying developing malfunctions before they cause severe disruptions or safety hazards is a primary engineering objective. However, modern computational fault diagnosis has historically been impeded by the proprietary nature of operational telemetry, maintenance logs, and failure records, which manufacturers and building operators rarely disclose due to commercial sensitivity and strict liability concerns. To address this persistent research barrier, investigators Xinda Wang, Haozhe Sun, Junjie Huang, and Qiang Zhang have introduced a standardized, publicly accessible benchmark published in the journal PLOS ONE. Their research establishes a comprehensive evaluation testbed specifically designed to assess whether contemporary computational architectures, including state-of-the-art large language models, can navigate the distinct operational tiers of elevator maintenance from direct sensor interpretation to structured root-cause deduction.
The benchmark addresses a structural disconnect within industrial diagnostic research, where previous public datasets have focused almost entirely on isolated signal processing tasks, such as vibration analysis in rotating machinery or generic anomaly detection across synthetic industrial time series. Real-world elevator troubleshooting rarely operates as a single isolated prediction; human maintenance engineers must systematically correlate physical sensor readings, consult formalized equipment taxonomies, and deduce failure mechanisms from incomplete or ambiguous symptom reports collected in the field. To capture this operational complexity, the authors structured their evaluation framework into three complementary, progressively demanding task groups labeled A, B, and C. Task Group A measures sensor-based fault detection and fine-grained classification; Task Group B assesses taxonomy-grounded category assignment and multi-cause identification; and Task Group C evaluates diagnostic reasoning when field evidence is constrained or deliberately missing. By standardizing input structures, formalizing closed-set output spaces, and decoupling scoring procedures across task boundaries, the benchmark prevents cascading error propagation and enables objective, reproducible comparisons across diverse algorithmic architectures.
In Task Group A, the investigation examines how models handle raw, serialized sensor observations drawn from a fixed, released evaluation repository containing 5,000 distinct operational records. This dataset encompasses telemetry from 247 anonymized elevator device identifiers across 57 anonymized equipment brands, incorporating 3,832 verified normal operational instances and 1,168 confirmed fault events distributed across seven distinct numerical failure codes. Subtask A1 poses a fundamental binary detection challenge, requiring models to determine strictly from numeric sensor snapshots whether an elevator is functioning normally or experiencing an active failure state. Subtask A2 demands a significantly more challenging multi-class prediction, isolating the 1,168 faulty records to evaluate whether an automated system can pinpoint the precise operational fault code among seven candidates. Because real-world elevator monitoring treats undetected failures as far more hazardous than false alarms, the researchers evaluated Subtask A1 using detection recall and a weighted F2-score that penalizes false negatives more heavily than false positives, alongside standard classification accuracy for Subtask A2.
Task Group B elevates the diagnostic challenge from raw numerical telemetry to domain knowledge grounding by evaluating how algorithms parse natural-language fault descriptions against authoritative regulatory standards. The researchers derived this module directly from the official elevator fault taxonomy promulgated by the China Special Equipment Safety and Energy Conservation Promotion Association, an established hierarchical framework comprising nine top-level failure categories and 47 fine-grained causative mechanisms. Subtask B1 presents 47 single-label classification items that map observed operational phenomena, such as unintended car movement, directly to one of the nine authoritative system categories. Subtask B2 introduces 500 multi-label diagnostic scenarios requiring models to identify the complete set of valid contributing causes corresponding to a diagnosed category from the 47 closed options. To assess model precision rigorously, the authors evaluated Subtask B1 using overall accuracy and macro-averaged F1-scores, while Subtask B2 was scored using both exact-match set accuracy and Jaccard similarity coefficients to distinguish complete structural correctness from partial set overlap.
Task Group C examines higher-order diagnostic inference under realistic field limitations where available symptom evidence is noisy, restricted, or partially absent. Constructing this module required building a specialized domain knowledge graph extracted from authoritative technical elevator service manuals, comprising four distinct node types covering fault categories, observed phenomena, underlying causes, and resolution or exclusion strategies. Subtask C1 specifically evaluates diagnostic resilience by presenting fault prediction instances where observable symptom descriptions are intentionally withheld; across the common core of 105 evaluated inputs, 95 contained deliberately omitted phenomena, representing a 35.8 percent symptom deletion rate. Subtask C2 formulates single-choice procedural decision problems where models select the most appropriate technical exclusion method across canonical problem stems. Subtask C3 requires multi-cause inference over constrained candidate lists derived from the verified knowledge graph. The scoring protocol measures missing-information accuracy and balanced macro-F1 for Subtask C1, decision accuracy for Subtask C2, and Jaccard similarity for Subtask C3, capturing whether models can maintain logical consistency when key diagnostic indicators are missing.
To establish baseline capabilities across contemporary artificial intelligence systems, the researchers conducted extensive empirical evaluations covering fourteen major language models, including claude-sonnet-4-5-20250929, deepseek-r1, deepseek-v3, gemini-2.5-pro, gemini-3-pro-preview, gpt-3.5-turbo, gpt-4-turbo, gpt-4o, gpt-4o-mini, gpt-5.2, qwen3-235b-a22b-instruct-2507, qwen3-8B, qwen3-max, and qwen3-next-80b-a3b-instruct. All models were tested under standardized zero-shot conditions without task-specific fine-tuning, applying uniform decoding parameters with temperature fixed at 0.6, top-p sampling at 0.95, and a generation ceiling of 16,384 tokens. For Subtask Group A, the team also trained conventional supervised reference baselines—including logistic regression, k-nearest neighbors with seven neighbors, and a shallow multilayer perceptron—using five-fold grouped cross-validation partitioned by anonymized device identifier. These classical baselines served as independent reference points for sensor interpretation rather than direct competitors within the language model zero-shot rankings.
The quantitative results demonstrated a pronounced, capability-dependent performance divergence across the multi-tier hierarchy, confirming that competence in one diagnostic phase does not guarantee effectiveness in another. In Task Group A, language models demonstrated considerable proficiency in coarse binary fault detection, with several systems achieving detection recall exceeding 80 percent on Subtask A1; notably, gpt-4o-mini recorded the highest detection recall at 93.24 percent and achieved an F2-score of 58.66. However, when required to distinguish between specific fault codes in Subtask A2, all evaluated language models exhibited substantial performance degradation, yielding accuracies between approximately 10 and 17 percent. The lightweight local model qwen3-8B achieved the highest A2 accuracy at 17.21 percent, underscoring the acute difficulty general-purpose language models experience when interpreting uncalibrated numeric sensor tables without domain-specific representation learning.
Conversely, models demonstrated vastly superior proficiency in Task Group B, where standardized domain taxonomies provided structured textual context for diagnostic reasoning. Several leading systems achieved category classification accuracies exceeding 70 percent on Subtask B1, with gemini-2.5-pro attaining the top performance at 78.72 percent accuracy and a balanced macro-F1 score of 78.90 percent. Non-parametric bootstrap resampling with 2,000 iterations established a two-sided 95 percent percentile confidence interval between 66.0 and 89.4 percent for its accuracy, reflecting the statistical distribution across the nine categorical classes. On Subtask B2, multi-label Jaccard similarity metrics confirmed that while advanced models frequently identified primary failure causes, their predictions often incorporated semantically plausible distractor options selected from related failure modes, illustrating the challenge of enforcing strict category boundaries during multi-label cause identification.
Evaluation across Task Group C revealed notable specialization among reasoning-focused architectures when managing missing or constrained operational data. On the common 105-item core of Subtask C1, gemini-3-pro-preview achieved the highest conditional missing-information accuracy at 73.68 percent, demonstrating superior stability when crucial symptom indicators were omitted. In the decision-making scenarios of Subtask C2, gemini-3-pro-preview shared the highest exact accuracy of 80.00 percent with gpt-3.5-turbo, while qwen3-235b-a22b-instruct-2507 led the constrained cause-inference evaluations of Subtask C3 with a Jaccard similarity score of 88.33 percent. While descriptive equal-weight composite summaries ranked gemini-2.5-pro highest overall at 59.61 points, followed by gemini-3-pro-preview at 58.13 and gpt-5.2 at 56.65, the researchers emphasized that these composite scores serve strictly as visual aids rather than measures of field utility.
The study authors emphasize critical methodological boundaries, noting that the benchmark serves as an experimental testbed for evaluating diagnostic-assistance methods rather than a validation framework for real-world operational deployment. The released sensor records necessarily lack physical calibration data, explicit engineering units, and time-disjoint splits, meaning model predictions cannot be interpreted as validated physical fault analyses or safety certifications. Furthermore, the canonical cores for Subtasks C2 and C3 comprise 15 unique problem stems each, resulting in wide confidence intervals that mandate cautious statistical interpretation. By open-sourcing the benchmark dataset, generation templates, and evaluation scripts through a public GitHub repository, the research team provides a reproducible foundation for developing hybrid diagnostic architectures that combine robust numerical sensor perception with grounded domain reasoning.
Subject of Research: Standardized multi-level benchmark for artificial intelligence evaluation in elevator fault diagnosis
Article Title: A benchmark for elevator fault diagnosis: Multi-level tasks from perception to diagnostic reasoning
Article References: Wang, X., Sun, H., Huang, J., & Zhang, Q. (2026). A benchmark for elevator fault diagnosis: Multi-level tasks from perception to diagnostic reasoning. PLOS One, 21(11), e0362972. https://doi.org/10.1371/journal.pone.0362972
Image Credits: AI Generated
DOI: 10.1371/journal.pone.0362972
Keywords: elevator fault diagnosis, benchmark dataset, large language models, cyber-physical systems, sensor data analysis, fault taxonomy, diagnostic reasoning, incomplete information, predictive metrics, closed-set classification, maintenance decision support, machine learning baselines, PLOS ONE, failure mode identification, root cause analysis, industrial artificial intelligence
Cite Scienmag News
Ophelia Keating. (November 21, 2026). Multi-Task Elevator Diagnostic Benchmark Reveals Language Model Strengths and Sensor Limitations. Scienmag. https://scienmag.com/multi-task-elevator-diagnostic-benchmark-reveals-language-model-strengths-and-sensor-limitations/
Ophelia Keating. "Multi-Task Elevator Diagnostic Benchmark Reveals Language Model Strengths and Sensor Limitations." Scienmag, 21 November 2026, https://scienmag.com/multi-task-elevator-diagnostic-benchmark-reveals-language-model-strengths-and-sensor-limitations/. Accessed 21 November 2026.
Ophelia Keating. "Multi-Task Elevator Diagnostic Benchmark Reveals Language Model Strengths and Sensor Limitations." Scienmag. November 21, 2026. https://scienmag.com/multi-task-elevator-diagnostic-benchmark-reveals-language-model-strengths-and-sensor-limitations/








