More Detail in AI Prompts Can Backfire When Chatbots Build Safety Models

Ask an AI chatbot to write a helmet-detection model with a vague one-line prompt, and you might expect a mediocre result. Ask it again with a dense, expert-level technical specification, and you might expect the model to get better. A new study shows that neither expectation holds in a simple way: for two of four frontier chatbots, the most detailed prompt produced syntactically perfect code that trained a neural network stuck at near-random guessing. The work, published in Discover Artificial Intelligence by Jorge Parraga-Alava of the Technical University of Manabí and Jorge Rodas-Silva of the State University of Milagro in Ecuador, offers one of the first controlled comparisons of how prompt design shapes the deep learning pipelines that large language models generate.

The motivation is occupational safety. According to the International Labour Organization, work-related accidents and diseases claim roughly 2.93 million lives annually, and hundreds of millions of workers sustain non-fatal injuries each year. Personal protective equipment such as helmets is the last line of defense against falling objects and machinery hazards, yet enforcement on factory floors and construction sites remains largely manual, intermittent, and error-prone. Computer vision offers an automated alternative: convolutional neural networks, the workhorse architecture of image recognition, can classify whether a worker is wearing a helmet. The bottleneck, the authors argue, is that designing effective CNN training pipelines still demands manual architectural decisions and exhaustive hyperparameter search, which are time-consuming and inconsistent across deployment contexts.

Large language models promise to remove that bottleneck by writing the training code themselves. But how should you prompt them? The researchers designed a hierarchical protocol with three tiers of increasing specificity. Tier 1 was a basic prompt that gave the model only a high-level task description and left every decision, from network backbone to learning rate schedule, to the AI. Tier 2 was a structured prompt that fixed a two-phase training scheme, first training only the classification head with frozen feature extractors and then fine-tuning the entire network, while asking the model to recommend a hyperparameter vector covering dropout rate, two learning rates, batch size, and the number of units in the dense classification layer. Tier 3 was a detailed prompt that prescribed everything: input dimensions of 224 by 224 by 3, a four-stage CNN with parallel 3-by-3 and 5-by-5 convolutional paths, explicit epoch counts and early-stopping patience values, a CosineAnnealingLR scheduler, and a fixed data augmentation policy including horizontal flips, rotations up to 15 degrees, color jitter, and random erasing.

Four chatbots, ChatGPT from OpenAI, Claude from Anthropic, Gemini from Google, and Grok from xAI, received all three prompts through their public web interfaces during a single collection week in February 2026, with no custom system prompts or decoding-parameter tweaks. That yielded twelve LLM-by-tier configurations. Each generated PyTorch script was executed exactly as delivered, with only a syntactic review for import resolution permitted, and each configuration was run twenty times with different random seeds on a MacBook Pro with an Apple M3 chip and 16 gigabytes of unified memory. No GPU was used, deliberately simulating the resource-constrained conditions of small and medium-sized industrial sites.

The dataset was purpose-built: 972 images scraped from publicly indexed web sources using queries in English and Spanish to promote geographic diversity, then filtered, deduplicated by cryptographic hash, restricted to single subjects, and standardized to 840-by-840 pixels on disk. Three independent annotators labeled the cranial region of each image as helmet or no helmet, achieving a Cohen’s Kappa of 0.93, almost perfect agreement. The final set was balanced at 486 images per class and split roughly 49 percent for training and 51 percent for testing, an unusually large evaluation share chosen to maximize the statistical reliability of the test estimates. A stratified five-fold cross-validation on the pooled data confirmed the held-out results, with mean accuracy of 0.989.

The headline result is a high-performance plateau. Ten of the twelve configurations converged to accuracy between 0.957 and 0.967 with AUC between 0.985 and 0.993, regardless of prompt tier. Claude at Tier 1 achieved the highest accuracy at 0.967 while also recording the lowest CPU usage at 12.2 percent, suggesting its internal priors about transfer learning are well aligned with the task. The LLM-generated pipelines matched, and in Claude’s case marginally exceeded, manually engineered baselines including ResNet50, EfficientNet-B0, and a YOLOv8-nano classifier evaluated under the same protocol. For this class of moderately sized binary tasks, the form of LLM involvement mattered less than whether training converged at all.

But accuracy told only half the story. A striking accuracy-calibration dissociation emerged at Tier 1: ChatGPT and Claude achieved high accuracy yet produced poorly calibrated probabilities, with test losses of 0.532 and 0.521, while Gemini and Grok delivered well-calibrated outputs with losses of 0.127 and 0.142. In a safety monitoring setting where downstream logic consumes predicted probabilities for threshold tuning and alert prioritization, that difference carries operational value even when accuracy is saturated. ChatGPT showed the only monotone loss improvement across tiers, falling from 0.532 to 0.197 to 0.135 as specificity increased, absorbing added detail as better calibration rather than better accuracy.

Then came the cliff. At Tier 3, Gemini and Grok suffered catastrophic convergence failures, with accuracies collapsing to 0.576 and 0.591, barely above the 0.5 chance level of the balanced binary task, and AUC degrading to 0.651 and 0.663. Their scripts were syntactically valid and executed end-to-end, so the failure was not about broken code. A per-script inspection revealed the mechanism: the two converging models, ChatGPT and Claude, silently substituted a pretrained ResNet18 backbone for the prescribed from-scratch multi-scale architecture, while the two failing models implemented the specification literally. A controlled ablation confirmed the causality, showing the from-scratch multi-scale design collapses to near chance under both CosineAnnealingLR and ReduceLROnPlateau, whereas a pretrained backbone converges under either scheduler. The destabilizing factor was the from-scratch architecture itself, not the scheduler or the prompt length.

A two-way ANOVA on accuracy found a large, highly significant LLM-by-tier interaction, with the interaction term of the same order as the main effects, meaning the effect of prompt tier cannot be interpreted independently of which chatbot received it. The authors evaluated three hypotheses and found the first two partially rejected: higher prompt tiers did not yield superior performance, and specificity did not consistently correlate with validity or performance gain, since executability remained 100 percent at every tier while the convergence success rate dropped from 100 percent to 50 percent at the most detailed tier. The third hypothesis, that LLM choice interacts with prompt tier, was fully supported. The authors caution that each LLM-by-tier cell was represented by a single generated script, so these observations describe the specific scripts obtained during the collection window rather than stable properties of the providers.

The practical recommendation is counterintuitive for prompt engineering folklore: for developing PPE compliance classifiers under similar conditions, basic or structured prompting is the recommended operating range, delivering competitive accuracy and full validity across all models without the convergence failures observed at the most prescriptive tier. The authors also outline mitigations for high-specificity scenarios, including smoke-train filtering of generated scripts, constraint relaxation, iterative re-prompting with loss feedback, and ensembling across chatbots. They emphasize that validation in real industrial settings, with multiple workers, occlusions, and variable lighting, remains necessary before these findings translate into deployed monitoring systems.

Subject of Research: Artificial Intelligence

Article Title: Effect of hierarchical prompt engineering on convolutional neural networks generated by large language models for helmet wearing compliance classification

Article References: Parraga-Alava, J., & Rodas-Silva, J. (2026). Effect of hierarchical prompt engineering on convolutional neural networks generated by large language models for helmet wearing compliance classification. Discover Artificial Intelligence, 6(1), Article 1534. https://doi.org/10.1007/s44163-026-02501-8

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02501-8

Keywords: prompt engineering, large language models, convolutional neural networks, helmet detection, personal protective equipment, PyTorch, code generation, occupational safety, transfer learning, model calibration, Effect, hierarchical