The Bigger AI Brain Is Not Always the Better Financial Reader
A new comparison of artificial-intelligence systems has challenged one of the most persistent assumptions in the current AI race: that a larger, general-purpose language model will automatically outperform a smaller system trained for a specific job. In a study published in Information Systems Frontiers, researchers tested GPT-3.5, the finance-specialized model FinBERT and a traditional dictionary-based method on a task that looks simple but is notoriously difficult for machines—determining whether corporate managers sound positive, neutral or negative in their annual reports. GPT-3.5 performed far better than the dictionary approach, but FinBERT achieved slightly higher accuracy, showing that scale and general versatility do not guarantee superior performance when language is tightly tied to a technical domain.
The target of the analysis was management sentiment in the “management discussion and analysis,” or MD&A, sections of corporate annual reports. These passages are written to explain a company’s financial results, operational conditions, risks and future prospects, and they are closely watched by investors and researchers. Yet managerial language rarely behaves like ordinary emotional prose. Words such as “liability,” “risk,” “challenge” or “uncertainty” may signal concern in everyday language but can be routine technical terms in accounting. A sentence that appears optimistic in isolation may also describe a worsening result relative to the previous year. Reliable sentiment analysis therefore requires more than counting upbeat or alarming words: it requires understanding how financial language functions within context.
The researchers assembled their dataset from annual reports issued by all A-share companies listed on China’s Shanghai and Shenzhen stock exchanges during the study period. The reports were collected through the WINGO database, converted from PDF files into machine-readable text and filtered to isolate MD&A narratives. Tables, headings, page furniture and other formatting artifacts were removed before the text was divided into individual sentences. Three trained research assistants independently labeled 6,000 randomly selected sentences as positive, neutral or negative. Disagreements were settled by majority vote. The annotation process produced a Fleiss’ kappa of 0.72, a level generally interpreted as substantial agreement, providing a human-labeled benchmark for testing the automated systems.
The first method was a conventional dictionary-based approach, which assigns sentiment according to preselected lists of positive and negative words. Such systems are fast, transparent and relatively inexpensive, but they can miss negation, context and the specialized meaning of financial vocabulary. FinBERT represents a more targeted form of machine learning. It is built on the BERT architecture, a transformer-based neural network that represents each word in relation to surrounding words, and it has been pretrained or fine-tuned on extensive financial material, including corporate disclosures, initial public offering prospectuses, corporate social-responsibility reports, internal-control reports and earnings conference calls. GPT-3.5, by contrast, is a general-purpose generative model. Rather than being retrained specifically for the task, it was guided through few-shot prompting, in which the researchers supplied examples of labeled sentences and asked the model to classify new ones.
The distinction matters because the three systems learn in fundamentally different ways. Dictionary methods use fixed lexical rules: if a sentence contains enough terms associated with positive or negative sentiment, its score shifts accordingly. FinBERT uses learned representations and a classification layer to estimate the probability of each sentiment class from the sequence of tokens. GPT-3.5 uses its broad language model to infer the classification rule from examples placed in the prompt. The researchers varied the number of examples, or “shots,” presented to GPT-3.5. Increasing the number of examples did not produce a simple, proportional improvement. Their experiments found that 51 examples produced the highest average accuracy in one comparison, while 24 examples offered a more stable performance and were selected for subsequent analyses because of lower volatility when accuracies were similar.
The headline result was a clear but non-linear ranking. GPT-3.5 substantially outperformed the dictionary-based method, demonstrating that a general-purpose model can extract meaningful managerial tone without being specialized for finance. But FinBERT performed slightly better than GPT-3.5 overall. The finding directly contradicts the idea that a larger model must dominate a smaller domain model simply because it has broader training and more parameters. In this setting, the advantage came from matching the model to the task. FinBERT’s exposure to financial documents appears to have helped it distinguish the difference between ordinary language and the carefully qualified, highly contextual language of corporate reporting.
The study also included an additional test of DeepSeek, a general-purpose model developed in a Chinese-language context. Using a standardized natural-language prompt, the researchers asked DeepSeek to classify each sentence in the same three-way scheme. DeepSeek reached an overall accuracy of 87.90 percent, narrowly exceeding GPT-3.5’s 87.20 percent. The two systems displayed different error patterns: DeepSeek had higher precision, meaning that its positive or negative predictions were more often correct, while GPT-3.5 had higher recall, meaning that it identified a larger share of the relevant positive and negative sentences. GPT-3.5 also achieved a higher F1 score for positive sentences and showed stronger recall for both positive and negative categories. Even so, both general-purpose systems remained less effective overall than FinBERT.
Those results suggest that the challenge was not simply a shortage of Chinese-language training data. If language coverage were the main obstacle, a model trained in a Chinese context might have been expected to close the gap decisively with the finance-specialized system. Instead, DeepSeek performed comparably to GPT-3.5, while both lagged behind FinBERT. The more plausible explanation is task-domain alignment: financial sentiment depends on conventions, terminology and implicit comparisons that general language competence does not always capture. A model can be fluent, knowledgeable and highly capable across many tasks while still making systematic mistakes when a narrow domain demands specialized interpretation.
To evaluate whether sentiment scores carried economic information, the researchers also connected the language classifications with company-level financial and market variables. They calculated tone as the percentage of positive sentences minus the percentage of negative sentences in each firm’s MD&A section, generating separate measures for FinBERT, GPT-3.5 and the dictionary method. The analysis considered outcomes including cumulative abnormal stock returns around the release of annual reports and future return on assets, alongside controls such as company size, leverage, book-to-market value, earnings, stock-return volatility, institutional ownership, firm age, unexpected earnings, accruals, report length, market listing and dividend announcements. Abnormal returns were estimated relative to a Fama–French five-factor model, allowing the researchers to examine whether language contained information beyond standard financial indicators.
The implications reach well beyond corporate reports. Sentiment analysis is increasingly used to study investor reactions, managerial communication, financial fraud, accounting practices and the likely effects of AI-assisted disclosure. A model that labels language incorrectly can distort these analyses, potentially turning neutral accounting language into apparent pessimism or interpreting cautious risk disclosure as a sign of corporate distress. The study does not argue that general-purpose LLMs are unsuitable for finance; GPT-3.5’s strong performance over dictionaries shows their considerable value. Instead, it offers a warning against treating model size as a universal proxy for intelligence. For specialized tasks, the winning system may be the one that has learned the right vocabulary, conventions and failure modes—not the one with the broadest claims or the largest architecture. The authors’ central message is increasingly relevant as companies and researchers deploy AI to read markets: performance depends on the fit between model and task, and that fit must be measured rather than assumed.
