GPT-4 beats fine-tuned BERT models at spotting names in legal contracts — but only when the workload is light

Legal contracts are dense thickets of names — people, companies, agencies, subsidiaries — and finding every one of them reliably has long been one of the most stubborn problems in legal technology. A new study from researchers at Ulster University and the law firm A&O Shearman, published in the journal Knowledge and Information Systems, has now put the hype to a hard test, pitting OpenAI’s GPT-4 against a battery of fine-tuned BERT-family models on a real legal entity recognition task — and then asking the question most academic papers avoid: which one is actually worth the money?

The task in question is named entity recognition, or NER, a foundational technique in natural language processing that involves locating and classifying mentions of things like persons and organisations within raw text. In the legal world, NER is the engine behind contract analysis, document redaction and large-scale information extraction. The trouble is that legal language is notoriously hostile to generic models: entities appear as abbreviations, acronyms and synonyms, and off-the-shelf tools trained on general English text stumble badly when confronted with contract boilerplate.

Traditionally, high-performing NER has required either hand-crafted rules or vast amounts of manually annotated training data — a bottleneck that is especially painful in law, where labelling must be done by qualified legal experts. The research team, led by Shefali Singhal and Niall McCarroll, took a different route. They repurposed the Contract Understanding Atticus Dataset (CUAD), a collection of 510 real contracts spanning 25 different contract types, into a new NER benchmark called CUAD-NER, with five legal domain experts labelling every PERSON and ORGANISATION entity — nearly 100,000 organisation mentions in total.

Against this dataset they lined up an unusually broad field of competitors: the industry-standard SpaCy pipeline, Microsoft’s Azure Entity Recognition V3 service, and four transformer models — DistilBERT, BERT base (cased), RoBERTa and DeBERTa — each fine-tuned on the legal contract data using standard low-rank adaptation (LoRA) settings on Azure’s machine learning platform. The challenger was GPT-4, deployed not with any fine-tuning at all, but with a carefully structured few-shot prompt that included ten worked examples written in a chain-of-thought style, adapted from the PromptNER strategy of Ashok and Lipton.

The prompt itself is a study in prompt engineering discipline. It contains three components: a definition section spelling out exactly what does and does not count as a PERSON or ORGANISATION entity, the contract text to be analysed, and the ten few-shot exemplars that force the model to reason step by step — deciding whether a given phrase is an entity, justifying the decision, and then assigning a label of ORG, PER or NA. The exemplars were chosen to give balanced coverage of positive identifications and negative rejections, and the team deliberately avoided optimising over example choice or ordering, aiming instead for the kind of reliable average-case performance a practitioner would actually deploy.

The results were striking. GPT-4 with few-shot chain-of-thought prompting achieved a Micro F1 score of 73 percent across both entity types — a five percentage point improvement over the best fine-tuned competitor, BERT base (cased), which scored 69 percent. RoBERTa managed 67 percent, while DistilBERT and DeBERTa each reached 66 percent. The generically trained baselines fared far worse: Azure Entity Recognition V3 scored just 53 percent, and SpaCy collapsed to a dismal 10 percent, mislabelling common nouns like

The five-point margin in favour of GPT-4 is notable not only for its size but for what it represents: a model that never saw a single labelled contract during training outperforming systems that were explicitly fine-tuned on the very same corpus. The fine-tuned transformers in the study were adapted using LoRA, a parameter-efficient technique that freezes the bulk of a pre-trained model’s weights and trains only small, low-rank update matrices. This keeps training costs manageable compared with full fine-tuning, but it still requires the annotated corpus, the compute infrastructure, and the engineering effort to run the adaptation — all of which the few-shot approach sidesteps entirely. For organisations that lack teams of legal experts available to annotate documents, that difference in workflow may matter as much as the difference in F1 score.

The Micro F1 metric used in the evaluation blends precision and recall into a single figure, and in an entity recognition task it treats every mention as equally important. That makes it a reasonable headline number, but the underlying trade-offs are worth considering. A redaction system, for instance, may tolerate false positives — flagging a phrase as an entity when it is not — far more readily than false negatives, which would leave sensitive names exposed in a document. Conversely, an information extraction pipeline feeding a contract analytics dashboard might prefer precision, since spurious entities clutter downstream analysis. The study’s aggregate scores do not by themselves reveal how each model distributes its errors between these two failure modes, and practitioners deploying such systems would likely want per-class breakdowns before committing to a choice.

The collapse of SpaCy to a 10 percent score is a reminder of how domain-specific legal text really is. SpaCy’s general-purpose pipeline relies on statistical models trained largely on news wire and web text, where entities tend to appear in predictable contexts — a person’s name adjacent to a title or a company name followed by corporate suffixes. Contract language inverts many of these expectations: defined terms appear in quotation marks, parties are referred to by capitalised shorthand after their first mention, and organisational names can be embedded deep within nested clauses. The result is a cautionary tale about assuming that tools which perform well on general text will transfer to specialised professional domains without substantial retraining.

The choice of CUAD as the foundation for the new benchmark also carries significance. CUAD was originally designed for contract clause classification — identifying which of 41 clause types appear in a given contract — rather than for token-level entity tagging. Converting it into CUAD-NER required the five legal experts to annotate every PERSON and ORGANISATION mention across roughly 100,000 organisation references, a labour-intensive process that itself illustrates the annotation bottleneck the paper seeks to address. The fact that such a dataset had to be created by hand underscores the economics at play: once a labelled corpus exists, fine-tuned models become viable; without one, few-shot prompting may be the only practical option.

The cost–benefit analysis that accompanies the performance comparison is arguably the study’s most distinctive contribution. Academic NER papers routinely report F1 scores but rarely model the financial realities of deployment, where inference costs scale with usage volume. The study distinguishes between low-usage and high-usage scenarios, recognising that the economics of calling a commercial LLM API differ fundamentally from hosting a fine-tuned open-weight model. At low volumes, the absence of training and infrastructure costs gives the few-shot approach an advantage; at high volumes, the per-token pricing of a commercial API can accumulate to the point where a self-hosted fine-tuned model, despite its lower accuracy, becomes the more economical choice. The crossover point between these regimes is exactly the kind of information practitioners need but rarely find in the literature.

It is also worth situating the findings within the broader trajectory of LLM capabilities. GPT-4’s performance here reflects a model evaluated without any task-specific adaptation, relying purely on in-context learning — the ability to infer a task’s requirements from examples embedded in the prompt. Chain-of-thought prompting, the technique used to elicit step-by-step reasoning before the model commits to a label, has been shown across many benchmarks to improve performance on tasks requiring deliberate judgement rather than pattern matching. Entity recognition in legal text sits somewhere between these categories: it involves both surface pattern recognition and contextual judgement about whether a phrase refers to a person or an organisation in a given contractual role. The structured reasoning the prompt enforces appears to help the model navigate this ambiguity more effectively than direct labelling would.

The study’s authors acknowledge that the few-shot approach is not without limitations. Performance on unfamiliar, domain-specific NER tasks had remained uncertain precisely because LLM behaviour on such tasks is sensitive to prompt design, and the team deliberately avoided optimising over example selection or ordering to reflect realistic deployment conditions. This choice favours robustness over peak performance, but it also means that a practitioner willing to invest in prompt refinement — selecting exemplars that better cover edge cases, or tuning the definition section — might achieve results beyond those reported. Conversely, the fine-tuned baselines were trained with standard LoRA settings rather than exhaustively hyperparameter-optimised, so neither side of the comparison represents a ceiling on what is achievable.

Looking forward, the authors suggest that few-shot chain-of-thought learning could be strengthened with additional support strategies, and that flexible prompting approaches are likely to prove especially valuable in low-resource scenarios where annotated data is scarce. The legal domain is a prime example: contracts are confidential, annotation requires expertise, and the diversity of contract types means that a model fine-tuned on one organisation’s documents may not generalise to another’s. In such settings, the ability to obtain competitive performance from a general-purpose model with nothing more than a well-crafted prompt and a handful of examples represents a genuine shift in the accessibility of legal NLP technology — one that could bring sophisticated document analysis within reach of smaller firms and legal aid organisations that could never afford the annotation campaigns traditional approaches demand.

The broader implication for the field is that the evaluation of NLP systems is maturing beyond accuracy metrics alone. By pairing performance figures with a structured cost analysis, this study offers a template for how future work might be framed: not as a simple leaderboard, but as a decision-support tool that acknowledges the real constraints under which organisations operate. As LLMs continue to improve and API pricing evolves, the balance between prompting-based and fine-tuning-based approaches will shift, but the framework for comparing them — performance, cost, and the practical realities of deployment — is likely to endure.

Subject of Research: Evaluating LLMs for named entity recognition: a cost-benefit analysis in the legal domain

Article Title: Evaluating LLMs for named entity recognition: a cost-benefit analysis in the legal domain

Article References: Evaluating LLMs for named entity recognition: a cost-benefit analysis in the legal domain. (n.d.). https://doi.org/10.1007/s10115-026-02886-z

Image Credits: AI Generated

DOI: 10.1007/s10115-026-02886-z

Keywords: Evaluating, LLMs, named, entity, recognition, cost-benefit, analysis, legal, domain, scientific research