Simpler Language Models Match Complex Architectures in Cyber Threat Entity Recognition

In an empirical benchmark evaluating named entity recognition for cyber threat intelligence, simpler encoder-based language models and classical statistical learners matched or outperformed intricate multi-layered architectures while requiring far less computational power. Led by researchers Maximilian Ranzinger, Florian Skopik, Dženan Hamzić, Max Landauer, Markus Wurzenberger, and Peter Knees, the study reviewed 130 publications and evaluated seven accessible implementations alongside large language model baselines across standardized security corpora. The evaluation demonstrated that additional architectural mechanisms, such as recurrent networks and multi-feature fusion modules, did not produce measurably superior entity extraction over streamlined models. Furthermore, the 95 percent confidence intervals of the top five evaluated architectures overlapped on the primary benchmark dataset, indicating that apparent differences among leading models remain statistically inconclusive under finite sample testing.

Cyber threat intelligence involves collecting, analyzing, enriching, summarising, and extracting cybersecurity data, including threat actor profiles, tactics, techniques, procedures, software vulnerabilities, and newly released defensive products. Security analysts face an overwhelming volume of unstructured text from open-source feeds, technical blogs, and incident reports, making manual tagging cost-prohibitive and impractical. Named entity recognition automates this pipeline by classifying unstructured text spans into categories such as vendors, threat groups, and vulnerability identifiers. However, applying these techniques to cybersecurity remains challenging due to specialized terminology, rapidly evolving vocabularies, and disparate annotation standards across reporting frameworks such as Structured Threat Information Expression domain objects. While recent academic trends favor increasingly complex deep learning pipelines and generative models, the computational overhead of deploying and retraining these architectures presents substantial operational hurdles for security operations centers.

To establish a comprehensive foundation, the research team conducted a systematic literature review querying prominent engineering and scientific databases including ACM Digital Library, IEEE Xplore, ScienceDirect, Scopus, Google Scholar, and Web of Science for works through January 28, 2026. The search mapped cybersecurity entity recognition across 130 distinct publications since the field emerged in 2013. The literature analysis revealed that deep learning components were incorporated in nearly all evaluated systems, with encoder-based language models derived from BERT dominating recent developments. Over half of the identified systems utilized Bidirectional Long Short-Term Memory networks to capture intra-sequence context, while Conditional Random Fields and Softmax classifiers served as primary decoders. Despite widespread interest in generative artificial intelligence, none of the published cybersecurity approaches had systematically tested reasoning-capable large language models.

From fifteen candidate publications providing available source code, eight were excluded after failing reproducibility checks: four lacked documentation, two relied on unmaintained dependencies like AllenNLP or older TensorFlow versions, one produced immediate runtime exceptions, and one omitted its core extraction module entirely. The seven surviving specialized architectures represented diverse design paradigms spanning twelve years of natural language processing development. These included STUCCO, a 2014 maximum entropy model utilizing averaged perceptrons; Twitter, a 2019 recurrent model coupling word2vec and character embeddings with a Conditional Random Field; CyNER, pairing RoBERTa embeddings with a Softmax head; BERT-CRF, connecting base BERT directly to a Conditional Random Field; Split-NER, an extractive question-answering framework; NER4CTI, integrating six embedding sources with attention mechanisms; and ESP, combining BERT embeddings with label token augmentation.

Alongside these seven specialized systems, the authors established prompt-based baselines using Google’s LangExtract framework to examine whether general world knowledge in large language models could rival task-specific training. The baselines utilized Gemma 3 with 27 billion parameters as a representative distilled model and DeepSeek-R1 with 32 billion parameters as an open-source reasoning model, executed locally on an NVIDIA A100 graphics processing unit with 40 gigabytes of memory. The benchmark utilized two primary datasets formatted under the Beginning-Inside-Outside tagging scheme: DNRTI, a corpus of 6,583 sentences and 23,404 entity mentions spanning 13 security categories, and STIXnet, a 52-document collection containing 910 sentences and 1,396 entity mentions mapping to 15 threat intelligence objects. Performance was measured across strict, exact, partial, and entity type boundary levels, while environmental impact was tracked using codecarbon.

On the primary DNRTI test corpus, specialized models delivered tightly clustered overall performance, with strict micro-averaged F1 scores spanning from 0.8068 to 0.8743. ESP attained the highest strict micro F1 score of 0.8743, followed closely by CyNER at 0.8689 and BERT-CRF at 0.8659. The decade-old statistical learner STUCCO secured an impressive strict micro F1 score of 0.8639, closely matching modern transformer pipelines and achieving the second-best exact boundary matching score of 0.8981. Split-NER achieved 0.8590, while the heavily engineered NER4CTI reached 0.8347. Twitter recorded a strict micro F1 of 0.8068, though its performance rose under relaxed entity-type scoring to 0.8564, showing reliable category assignment despite looser boundaries. Across all models, macro-averaged F1 scores exceeded micro-averaged scores, demonstrating that specialized architectures did not disproportionately favor majority classes.

In stark contrast to tailored cybersecurity models, both prompt-based large language model baselines exhibited substantial performance deficits across all evaluation dimensions. Gemma 3 recorded a strict micro F1 score of 0.4849, while the reasoning-enabled DeepSeek-R1 achieved only 0.4514. Both baselines fell more than 30 percentage points below every specialized cybersecurity system on the DNRTI test set, well outside the bounds of sampling error. Moreover, the addition of explicit reasoning chains in DeepSeek-R1 provided no empirical benefit over the non-reasoning Gemma 3 model, yielding overlapping confidence intervals. The baselines showed slight competitiveness on a narrow subset of low-frequency categories where world knowledge compensated for zero task training, such as Area, Exp, and Hacker Organization. However, baseline extraction accuracy collapsed on multi-word entities, dropping toward near-random performance when span lengths reached four or more words.

Through an entity attribute analysis bucketing instances by length, sentence length, entity density, out-of-vocabulary density, frequency, and label consistency, the authors identified key determinants of model behavior. Sentence length had negligible influence across all models, whereas entity density moderately improved baseline extraction by providing richer contextual structures. Supervised models degraded as out-of-vocabulary density increased, with NER4CTI demonstrating acute vulnerability to unseen vocabulary. Crucially, entity consistency and word consistency emerged as the strongest predictors of recognition accuracy. When entity strings appeared in testing with labels never observed during training, model performance plummeted by 40 to 76 percentage points. Even top-ranking systems like ESP experienced F1 drops from 0.87 down to 0.56 under severe label inconsistency, demonstrating that ambiguous or unstable labeling cannot be resolved through textual context alone.

To evaluate whether broader textual context enhances entity recognition, the team tested models on the STIXnet dataset by comparing individual sentence processing against whole-document ingestion. Contrary to common theoretical assumptions that extended context resolves coreference and contextual ambiguities, joint whole-document processing degraded performance across nearly all architectures. The prompt-based baselines suffered severe collapses, with Gemma 3 dropping 24 percentage points and DeepSeek-R1 plunging 35 percentage points in strict micro F1 score. Among specialized models, STUCCO and NER4CTI both declined by 22 percentage points, while BERT-CRF and Split-NER dropped by more than 10 percentage points. Twitter was largely resilient, losing just over one percentage point, while CyNER was the sole model to register an improvement, gaining a modest 1.5 percentage points. The researchers concluded that longer input windows introduced extraneous noise that diluted attention signals.

Operational cost evaluations exposed massive disparities in processing speed and energy consumption, establishing distinct Pareto frontiers for training and inference. For inference on DNRTI, STUCCO was the fastest system by an extraordinary margin, processing each sentence in just 0.932 milliseconds while consuming 0.231 milliwatt-hours. ESP achieved the second-fastest inference speed at 3.846 milliseconds per sentence and consumed the lowest inference energy of all systems at 0.153 milliwatt-hours per sentence, making it the sole energy-optimal model on the Pareto frontier. Conversely, BERT-CRF required 102.966 milliseconds per sentence, making it roughly 110 times slower than STUCCO. Prompted large language models proved impractical for high-throughput deployment, taking over five seconds per sentence while consuming between 329.556 and 499.285 milliwatt-hours—representing energy overheads 88 to 133 times higher than the slowest specialized model.

The training efficiency analysis further underscored the severe penalties associated with architectural over-engineering. STUCCO completed full training on DNRTI in just 210.720 seconds, roughly three and a half minutes, consuming a mere 6.833 watt-hours of electrical energy. Twitter followed at 249.318 seconds and 12.381 watt-hours, demonstrating that lightweight recurrent pipelines can be retrained rapidly. BERT-CRF trained in 1,825.641 seconds, approximately 30 minutes, using 123.632 watt-hours, presenting a balanced trade-off between modern transformer representations and computational cost. ESP required 3,347.259 seconds, about 56 minutes, consuming 215.588 watt-hours. In contrast, heavily engineered NER4CTI demanded 29,612.061 seconds—over eight hours of GPU execution—and consumed 2,622.717 watt-hours, exceeding CyNER’s energy consumption by more than fourfold. This enormous computational expenditure yielded no performance advantage, placing STUCCO, BERT-CRF, and ESP on the optimal training frontier.

A crucial insight emerging from the benchmark is that cybersecurity entity recognition models depend heavily on memorizing training vocabularies rather than exhibiting generalized reasoning. When evaluating entities that never appeared in the training set, model performance plummeted by approximately 30 percentage points across all seven supervised models. Standard academic evaluation protocols that stratify dataset splits purely by entity type mask this vulnerability because identical entity strings recur across training and test partitions. The authors emphasize that operational security deployments should prioritize consistent annotation guidelines and high-quality domain corpora rather than pursuing complex architectural enhancements. For production security operations centers handling massive real-time intelligence feeds, streamlined encoder-based models paired with simple Softmax or Conditional Random Field decoders offer the most practical compromise, while vectorized classical models like STUCCO excel when minimal latency and frequent retraining are essential.

Subject of Research: Benchmarking named entity recognition architectures for cyber threat intelligence

Article Title: A systematic comparison of Named Entity Recognition approaches for Cyber Threat Intelligence

Article References: Ranzinger, M., Skopik, F., Hamzić, D., Landauer, M., Wurzenberger, M., & Knees, P. (2026). A systematic comparison of Named Entity Recognition approaches for Cyber Threat Intelligence. Machine Learning with Applications, 26, Article 101064. https://doi.org/10.1016/j.mlwa.2026.101064

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101064

Keywords: cyber threat intelligence, named entity recognition, information extraction, natural language processing, language models, BERT, RoBERTa, STUCCO, Conditional Random Fields, DNRTI dataset, STIXnet, DeepSeek-R1, Gemma 3, energy consumption, inference latency, sequence labeling, reproducibility