Small On-Device AI Model Trails Cloud Giant on Building Engineering Questions

A compact artificial intelligence model that runs entirely on a single workstation matched one of the world’s most capable cloud AI systems on a short set of basic building engineering questions, but fell well behind when experts judged its longer written answers. That is the central finding of a preliminary case study published in Architectural Research, which examined EXAONE 4.0 1.2B, an open-weight on-device large language model released by LG AI Research, against ChatGPT-4o as a high-capability cloud reference. The study, led by Chang-Heon Cheong and Jeong-Hoon Lee, evaluated ten Korean-language items drawn from the level of the national qualification examination taken by undergraduate architectural engineering students in the Republic of Korea, covering indoor environmental quality, energy and post-occupancy evaluation.

The distinction between the two systems matters for how the results should be read. On-device models execute locally on a personal computer or edge device, which can reduce dependence on network connectivity and keep prompts and locally stored documents within the user’s own computing environment, but their smaller scale can constrain performance. EXAONE 4.0 1.2B has approximately 1.07 billion non-embedding parameters, while the architecture and parameter count of ChatGPT-4o are not publicly disclosed. The authors stress throughout that ChatGPT-4o served as a reference point rather than a size-matched competitor, and that the observed differences cannot be causally attributed to parameter count alone, since the systems also differ in training, alignment, inference infrastructure and deployment constraints.

The item set contained four multiple-choice questions, three single-step calculation problems and three short-answer questions. Items one through seven were adapted from past architectural-engineering examination material made available through Q-Net, while items eight through ten were researcher-designed to elicit extended explanations. Both models received the same Korean question content under the same simple, minimal answer-request format, with no model-specific prompt optimization. The authors emphasize that this standardization controls a major source of avoidable variation but does not establish the maximum performance attainable from either model, because prompt design can materially affect large language model performance and no systematic prompt-sensitivity experiment was conducted.

On the objective items, the outcome was a clean sweep for both systems. Both models answered all four multiple-choice questions and all three calculation items correctly. The calculation problems were deliberately elementary: each required one principal relation and straightforward arithmetic, with all inputs supplied in consistent units, no code-table lookup and no multi-stage dependency. The authors caution that seven correct answers from seven items are far too few to estimate broad domain accuracy, and that such problems do not represent the multi-variable, code-referenced calculations used in professional building-environment practice, such as heating or cooling load calculations that chain together surface schedules, orientation-dependent solar gains and design conditions retrieved from standard tables.

The reasoning traces behind the correct answers revealed a telling difference. On a thermal-environment item, EXAONE translated Korean terms into English during its reasoning and conflated equivalent temperature, a standardized index, with a related thermal-comfort term from the older Bedford formulation, which represents a distinct historical index, before returning to Korean and selecting the correct option. The authors describe this as consistent with a terminology or knowledge-coverage limitation, while noting that the experiment cannot determine whether the cause lies in training coverage, translation behavior, tokenization or another model-internal factor. On a glare question, by contrast, EXAONE added a qualification that source size and luminance distribution can affect perceived glare, a more nuanced justification than the conventional textbook explanation used by ChatGPT-4o, though a single response supports no broader claim.

For the three open-ended items, five doctoral-level experts in building environmental engineering rated each model’s responses on five-point scales for accuracy, completeness and logical consistency, yielding nine rated units and 45 individual ratings per model. The overall mean was 4.18 for ChatGPT-4o and 2.98 for EXAONE, a difference of 1.20 points, with ChatGPT-4o scoring higher in every one of the nine item-by-criterion cells. The qualitative pattern was more nuanced than the numbers suggest. ChatGPT-4o more consistently organized its answers around concepts and the requested scope, while EXAONE tended to supply specific technical, operational and Korean-regulatory content, naming materials such as vacuum insulation panels, aerogel and building-integrated photovoltaics and citing the Korean Building Energy Saving Design Standards, but occasionally omitted canonical elements, expanded beyond the requested scope or added unsupported detail.

One EXAONE response illustrated a practical risk of confident but unsourced specificity. In answering a question on preventing indoor airborne infection transmission, the model recommended replacing air-cleaner filters two to three times daily without identifying any product, standard or operating condition that would justify such an interval, and placed a carbon dioxide value and a fine particulate matter value side by side as alert thresholds without explaining their different purposes or averaging periods. The authors argue this failure mode is more consequential than simple verbosity, because a non-specialist user may not recognize which numerical details require verification. The response also drifted from airborne infection control toward general indoor air quality management and contained malformed numbering.

The expert ratings themselves demand cautious interpretation. Fleiss’ kappa, which measures exact categorical agreement among multiple raters corrected for chance, was just 0.020 for ChatGPT-4o and 0.044 for EXAONE, indicating only slight agreement beyond chance. Free-marginal kappa, which fixes expected chance agreement at a fixed level, was 0.333 and 0.222, falling in the fair rather than moderate range. Exact pairwise agreement among the five experts was 46.7 percent and 37.8 percent respectively, though agreement within one scale point reached 93.3 percent and 77.8 percent, showing that many disagreements involved neighboring Likert categories. The authors therefore treat the mean scores as descriptive summaries rather than calibrated measurements, and note that disagreements may partly reflect how experts penalized technically relevant but out-of-scope elaboration.

The study’s boundaries are explicit. Ten items, one on-device model and one cloud reference do not constitute a representative sample, the analysis rests on a single recorded response per model-item condition, and no field deployment measured latency, memory use or energy consumption on portable hardware. Within those limits, the authors conclude that the results support further investigation of on-device language models as supervised offline assistants in settings where connectivity is intermittent or external data transmission is restricted, such as construction sites, security-restricted facilities and classroom training, but not as unsupervised tools for engineering calculation or compliance decisions. They propose a tiered calculation battery, size-matched baselines, prompt-sensitivity testing and a field study on actual deployment hardware as the next steps toward a confirmatory evaluation.

Subject of Research: Evaluation of an on-device large language model's knowledge accuracy and response characteristics in building environmental engineering

Article Title: Knowledge Accuracy and Response Characteristics of an On-Device Large Language Model in Building Environmental Engineering: a Preliminary Case Study of EXAONE 4.0 1.2B Using ChatGPT-4o as a Cloud Reference

Article References: Cheong, C.-H., & Lee, J.-H. (2026). Knowledge Accuracy and Response Characteristics of an On-Device Large Language Model in Building Environmental Engineering: a Preliminary Case Study of EXAONE 4.0 1.2B Using ChatGPT-4o as a Cloud Reference. Architectural Research, 28(1), Article 17. https://doi.org/10.1007/s44515-026-00018-w

Image Credits: AI Generated

DOI: 10.1007/s44515-026-00018-w

Keywords: large language models, on-device AI, EXAONE 4.0 1.2B, ChatGPT-4o, building environmental engineering, expert evaluation, inter-rater agreement, Korean-language benchmark, offline AI assistants, indoor environmental quality, building energy, preliminary case study