AI Reads the Boardroom: Can Language Models Measure Digital Transformation?

Every year, thousands of companies declare in their annual reports that they are embracing digital transformation. Cloud computing, artificial intelligence, big data platforms, and intelligent manufacturing appear in corporate disclosures with increasing frequency, yet turning those words into a rigorous, comparable scientific measurement has long been one of the most stubborn problems in empirical economics and management research. A new study published in the Journal of Big Data by Zhenkun Zhou, Yihan He, and Tao Ren of Capital University of Economics and Business in Beijing tackles this challenge head-on, asking a deceptively simple question: can large language models, the same technology behind conversational AI assistants, reliably read corporate texts and judge how digitalized a company really is?

The stakes are higher than they might appear. Digital transformation is widely credited with reshaping business models, operational efficiency, and competitive advantage, but researchers who want to test these claims need a way to quantify the phenomenon across thousands of firms and many years. Traditional approaches have leaned on hand-built dictionaries of digital keywords or on conventional machine learning classifiers trained on labeled examples. Both methods struggle with the messy reality of corporate language, in which the same term can signal a genuine technological capability, a vague aspiration, or a marketing flourish. The Chinese research team argues that the contextual sensitivity of modern language models makes them natural candidates for the job, and their results suggest the field may be on the verge of a methodological shift.

At the heart of the study is a multi-indicator, multi-level text measurement framework. Rather than collapsing digital transformation into a single score, the framework distinguishes two dimensions. The first is digitalization, which captures whether and how a firm applies specific digital technologies. The second is effectiveness, which probes whether those applications actually produce measurable results for the business. This separation matters because a company can boast extensive technology adoption while showing little evidence that the adoption has changed its operations or performance. By classifying enterprise disclosures along both dimensions, the framework aims to give researchers a finer-grained instrument than a simple keyword count.

To find out how well large language models perform this task, the authors designed a three-dimensional experimental architecture that is as much about the science of prompting as it is about digital transformation itself. The first dimension concerns output modes: should a model produce all indicator classifications in a single synchronous response, or should it assess each indicator one at a time? The second dimension covers prompting strategies, including zero-shot instructions with no worked examples and chain-of-thought prompting, which asks the model to reason step by step before answering. The third dimension is fine-tuning, in which a base model is further trained on task-specific labeled data. By varying each factor systematically, the study maps how these engineering choices interact with model capability and task difficulty.

The headline finding is striking: large language models generally outperform traditional machine learning baselines even in zero-shot settings, meaning they beat classifiers that required task-specific training without receiving any task-specific examples themselves. This is a meaningful result for anyone who has assumed that bespoke supervised models remain the gold standard for text classification in social science. The generality of pretrained language models, it seems, transfers remarkably well to the specialized vocabulary and rhetorical conventions of corporate disclosure, at least for indicators that are explicitly stated in the text.

The experimental dimensions produced more nuanced lessons. Synchronous multi-indicator output, in which the model evaluates several indicators within one prompt, generally performed better than asking the model to produce one classification per response. The authors suggest that assessing indicators together gives the model a richer picture of the firm’s overall digital posture, allowing each judgment to inform the others. Chain-of-thought prompting and fine-tuning, by contrast, showed effects that varied across models and tasks. Neither technique was a universal upgrade; in some configurations they helped, while in others they added little or even hurt. For practitioners, this is a caution against assuming that the latest prompting tricks or a round of fine-tuning will automatically improve any given measurement task.

Model identity also mattered. Performance varied across specific models and task settings, with both recent general-purpose models and reasoning-oriented models showing competitive results in different scenarios. In other words, there is no single champion model for this kind of work; the best choice depends on the indicator being measured and the configuration being used. This heterogeneity is itself informative, because it implies that researchers adopting language models for text-based measurement should benchmark several models and output configurations rather than defaulting to whatever model is currently most popular.

Perhaps the most scientifically interesting result concerns the gap between the two dimensions of the framework. The models performed well on explicit digital technology indicators, where the relevant evidence is directly stated in the text, but showed weaker performance on digital effectiveness indicators, which require contextual inference about whether technology adoption has translated into real business outcomes. This distinction between extraction and inference is a fundamental property of how language models process text. Recognizing a mention of cloud computing is a pattern-matching task that modern models handle with ease; judging from surrounding narrative whether that adoption has improved profitability or operational resilience demands a deeper synthesis of context, and the study shows that current models are not yet equally strong at it.

The authors did not stop at a single set of experiments. To guard against the possibility that their findings were an artifact of one language or one family of models, they conducted additional validation with non-Chinese multilingual language models and carried out significance analysis of the performance differences. These robustness checks supported the main conclusions, strengthening the case that the advantages of large language models over traditional machine learning are general rather than specific to the Chinese corporate disclosure corpus on which the framework was developed. The work was supported by the National Natural Science Foundation of China under grant number 62302319, and the article is published open access, making the methodological details available to any research group that wants to replicate or extend the approach.

The implications reach well beyond academic measurement. Regulators, investors, and analysts increasingly scrutinize corporate claims about digital capability, and text-based indicators built from public disclosures are becoming standard tools in finance and management research. If large language models can classify these disclosures more accurately and more cheaply than manual coding or bespoke classifiers, the cost of large-scale empirical studies could fall dramatically, and the set of answerable research questions could expand accordingly. At the same time, the study’s own results counsel humility: effectiveness judgments, the ones closest to what stakeholders actually care about, remain the hardest for machines to make. The Beijing team’s framework offers both a practical tool and an honest map of where the technology currently succeeds and where human judgment, or future model improvements, will still be needed. As enterprises continue to fill their reports with the language of digital change, the task of telling substance from slogan may increasingly belong to machines, provided their human overseers know exactly which questions to ask and how to ask them.

Subject of Research: Using large language models to measure enterprise digital transformation from corporate disclosure texts

Article Title: Can large language models assess enterprise digital transformation?

Article References: Zhou, Z., He, Y., & Ren, T. (2026). Can large language models assess enterprise digital transformation?. Journal of Big Data. https://doi.org/10.1186/s40537-026-01573-8

Image Credits: AI Generated

DOI: 10.1186/s40537-026-01573-8

Keywords: large language models, enterprise digital transformation, text measurement, machine learning, chain-of-thought prompting, fine-tuning, corporate disclosure, digitalization, prompt engineering, natural language processing, Journal of Big Data, zero-shot learning