Hidden in Plain Prompt: How a Subtle Design Flaw Leaks Database Secrets to AI Models

Credit rating agencies sit at one of the most sensitive intersections of the modern financial system. Their assessments move capital markets, price sovereign debt, and shape systemic risk judgments, all of which depend on access to deeply confidential corporate financial data. It is little wonder, then, that these institutions face a painful dilemma as large language models sweep through the business world: the same AI tools that promise dramatic productivity gains also threaten to transmit regulated, non-public information straight to external cloud services. In several jurisdictions, including South Korea’s financial sector and the European Union under GDPR Article 44 and the forthcoming AI Act, network segregation rules effectively prohibit internal systems from calling external AI endpoints at all. A new study published in Discover Artificial Intelligence by Munil Yang of the Institute for Industrial Policy Studies in Seoul now offers one of the first rigorous, empirical examinations of whether a popular technical fix for this dilemma actually works.

The dominant way organizations let AI query their databases is a technique called Text-to-SQL. The language model receives the full schema definition of the database, including every table and column name, and then writes SQL queries directly against it. The problem is that this approach hands the model everything: cryptic column identifiers, proprietary encoding schemes, and business logic embedded in the schema itself. In a credit rating context, transmitting such details to an external API endpoint constitutes exactly the kind of cross-boundary data transfer that network segregation laws are designed to prevent. The proposed alternative is a Semantic Layer, an abstraction interface borrowed from business intelligence architecture that sits between the model and the database, exposing only curated business concepts such as dimensions and measures while supposedly keeping raw schema details hidden inside the organization.

The intellectual pedigree of this idea traces back to David Parnas’s celebrated 1972 principle of Information Hiding, which holds that software modules should conceal their internal implementation details behind minimal, stable interfaces. In theory, a Semantic Layer instantiates this principle at the AI boundary: the model sees concepts like region and average salary, never the underlying physical columns they map to. But Yang’s study delivers a striking and cautionary finding. Whether the abstraction actually hides anything depends on an implementation detail that a schema-level analysis alone cannot reveal: whether the mapping from business concepts back to physical SQL is itself included in the prompt sent to the model.

The initial implementation examined in the study, labeled v1, looked entirely reasonable on paper. Its prompt contained elegant, human-readable concept definitions in which raw identifiers like A3 and A11 were replaced with meaningful names. But the same prompt also included a SQL mapping reference translating each concept back to its physical expression, along with a listing of physical table and column names needed to compose valid joins. When Yang performed an exhaustive verification of the complete payload actually transmitted to the model, checking every token across all benchmark questions, the result was damning: all fifteen sensitive column identifiers that the concept layer was nominally designed to hide were present in the prompt. Measured at the payload level, v1 achieved precisely zero reduction in schema exposure compared with raw Text-to-SQL.

The corrected implementation, v2, takes a fundamentally different approach. The model receives only concept names, their descriptions, and concept-level join relationships, and is instructed to respond using abstract concept references rather than physical SQL. A local compiler, running inside the organization and never exposed to the model, resolves those references into physical SQL after the API response returns. Verified exhaustively across the full benchmark, v2 removes all fifteen designated schema identifiers from the LLM-facing payload. The lesson Yang draws is pointed: information hiding at an AI interface is achieved by the complete prompt an implementation sends, not by the presence of concept-level names somewhere within it, and verifying it requires payload-level auditing rather than inspection of the abstraction layer’s design in isolation.

What about accuracy, the other half of the dilemma? Here the results are more nuanced and, in places, genuinely surprising. The study evaluated three conditions, Text-to-SQL, Semantic Layer v1, and v2, across three model tiers spanning a capability range, on two benchmarks. On a purpose-built credit-rating benchmark of sixty questions across six credit risk categories, designed alongside the Semantic Layer specification itself, v2’s accuracy improved steadily with model capability, rising from 38.7 percent at the weakest tier to 60.7 percent at the strongest, where it was numerically the highest of the three conditions, ahead of v1 at 58.7 percent and Text-to-SQL at 52.0 percent. Yang is careful to note that with only sixty questions, this strongest-tier advantage does not reach statistical significance, so the corrected architecture should be read as matching, not definitively beating, its baselines while additionally achieving full schema-identifier removal.

The picture changes dramatically on an externally authored benchmark, the financial subset of the well-known BIRD dataset, evaluated against the same underlying Czech banking schema. There, v2 trailed both baselines at every single model tier, and the gap between the in-specification benchmark and the external one widened as model capability increased. Yang attributes this deficit largely to genuine limitations of the v2 compiler and specification when confronted with question types they were never designed to handle, rather than to simple implementation defects. The author names this pattern selective encapsulation: some concepts benefit enormously from abstraction, particularly those with opaque encodings like single-character loan status codes, while others, such as already-transparent date fields, gain nothing from an extra layer of indirection. Crucially, it is presented as an empirical design lesson rather than a predictive theory.

The failure analysis contains perhaps the most operationally consequential insight of the entire study. At the weakest model tier, nearly half of v2’s outputs failed to compile or execute at all, a highly visible failure mode that falls steeply as capability rises, dropping to just 11 percent at the strongest tier. But the rate of wrong-but-executable results, queries that run successfully yet return incorrect answers, moved in the opposite direction, reaching over a third of generations at the stronger tiers. Unlike compile errors, these silent failures cannot be caught automatically before a result reaches an analyst. In other words, upgrading to a more capable model reduces the visible failure rate while leaving, or even increasing, the invisible one, a counterintuitive trap for any organization deploying these systems in high-stakes settings.

A component ablation added further practical texture. Removing few-shot examples from the v2 prompt caused a significant accuracy decline and nearly doubled the compile-error rate, while removing natural-language business descriptions made no significant difference. The few-shot examples, not the richness of the concept descriptions, emerged as the primary driver of the corrected architecture’s accuracy. Yang also withdrew two claims from the original submission: a purported theoretical extension of Information Hiding, which merely restated a standard engineering requirement, and a characterization of the results as evidence for the so-called Jagged Frontier of AI capabilities, since a smooth decline in accuracy with task difficulty does not establish the irregular capability boundary that concept actually describes.

The study’s practical message for regulated financial institutions is deliberately conditional rather than promotional. A correctly implemented Semantic Layer can achieve full schema-level information hiding without a detectable aggregate accuracy cost, but only when the specification is designed and maintained against the target query workload, and the benefit does not transfer automatically to new question distributions, even on the same database. Yang recommends treating specification design as an ongoing engineering task, including a task-type audit of which query categories are adequately covered, prioritizing compiler engineering for the join-path resolution failures that persist across all model tiers, and establishing human-review protocols specifically aimed at catching wrong-but-executable results. To enable independent scrutiny, the author has released the first domain-specific natural-language-to-SQL benchmark for credit rating contexts, complete with gold queries and an execution-based self-check, alongside all code and consolidated results. Whether these findings, obtained on a single schema with models from a single vendor, extend to proprietary rating databases and other model families remains an open question, but the core warning stands: at the AI boundary, security is not what your architecture looks like, it is what your payload actually contains.

Subject of Research: Semantic Layer architectures for reducing schema identifier exposure in LLM-based database querying in credit rating contexts

Article Title: A semantic layer architecture for removing schema identifiers from external LLM prompts, with model-dependent accuracy trade-offs

Article References: Yang, M. (2026). A semantic layer architecture for removing schema identifiers from external LLM prompts, with model-dependent accuracy trade-offs. Discover Artificial Intelligence, 6(1), Article 1374. https://doi.org/10.1007/s44163-026-02147-6

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02147-6

Keywords: semantic layer, large language models, Text-to-SQL, information hiding, schema identifier exposure, credit rating agencies, data security, selective encapsulation, BIRD benchmark, prompt architecture, model-dependent accuracy, AI compliance