Thursday, November 19, 2026
Science
No Result
View All Result
  • Login
  • HOME
  • SCIENCE NEWS
  • CONTACT US
  • HOME
  • SCIENCE NEWS
  • CONTACT US
No Result
View All Result
Scienmag
No Result
View All Result
Home Science News

Visual Inputs Expose Multimodal Vision-Language Models to Hidden Adversarial Vulnerabilities

November 19, 2026
in Science News
Blake Davidson
By Blake Davidson Scienmag Editorial Profile - Data Science
Reading Time: 6 mins read
0
Visual Inputs Expose Multimodal Vision-Language Models to Hidden Adversarial Vulnerabilities

Visual Inputs Expose Multimodal Vision-Language Models to Hidden Adversarial Vulnerabilities

65
SHARES
587
VIEWS
Share on FacebookShare on Twitter
ADVERTISEMENT

Large vision-language models that process both images and text remain vulnerable to cross-modal adversarial exploitation because their underlying architectures encode latent semantic information from visual inputs even when explicit optical character recognition fails or safety filters attempt to block malicious content. A systematic representational analysis and evaluation study conducted by researchers Juan Ren, Mark Dras, and Usman Naseem reveals that integrating visual inputs significantly expands the attack surface of multimodal artificial intelligence systems, allowing carefully crafted adversarial inputs to bypass safety mechanisms that were originally designed for text-based models.

The study examines why cross-modal adversarial attacks succeed across leading open-source multimodal systems, including LLaVA-1.5, LLaMA-3.2-Vision, Qwen-2.5-VL, and InternVL-2.5. Early generations of large vision-language models primarily inherited their safety capabilities from base language models without safety-aware designs embedded directly in the vision-language interface. Although newer architectures introduce post-fusion safety alignment, vulnerabilities persist because continuous visual embeddings can be manipulated through gradient perturbations or visual formatting, shifting continuous representations past textual moderation filters while leaving human-perceived meaning intact.

To evaluate these security dynamics, the researchers established a taxonomy categorizing multimodal adversarial attacks into five distinct operational types. Type I attacks involve gradient-based adversarial perturbations applied directly to pixel values of images. Type II attacks render harmful instructions directly into typographic images to evade text moderation, as exemplified by the FigStep framework. Type III attacks separate harmful intent across modalities by assigning action verbs to text prompts and target nouns to visual inputs, as seen in MMSafetyBench. Type IV attacks disguise malicious intent within subtle interactions between seemingly benign text and images, as evaluated in SIUO. Finally, Type V attacks form hybrid ensembles combining multiple perturbation and encoding strategies.

A central finding from the team’s representational analysis centers on how models decode visual inputs in the latent token space when confronted with typographic adversarial attacks like FigStep. By projecting intermediate image embeddings through projection matrices into the language model vocabulary space using cosine similarity mapping, the researchers decoded the semantic tokens captured by the models. In LLaVA-1.5, which lacks dedicated optical character recognition modules, the model failed to decode exact written words in open-ended queries but nevertheless projected visual patch embeddings directly to semantically loaded tokens such as method, illegal, and organ within a twenty-four by twenty-four grid matching its latent architecture.

When examining LLaMA-3.2-Vision, which incorporates optical character recognition capabilities, the researchers observed a different behavioral pattern driven by aggressive safety alignment. The model consistently refused both open-ended and multiple-choice character recognition queries involving typographic adversarial images, frequently outputting generic refusals asserting that it could not provide personal identity information even when images contained no identity-related content. However, latent analysis of patch-wise token projections across its extensive vocabulary of over one hundred and ten thousand tokens revealed that the underlying embeddings still captured semantically relevant fragments, demonstrating that latent semantic encoding persists beneath surface-level refusals.

In contrast, models equipped with advanced optical character recognition and tighter vision-language fusion, such as Qwen-2.5-VL and InternVL-2.5, accurately extracted text from typographic images and correctly answered open-ended queries. This architectural capability substantially strengthened their resistance against simple visual-textual obfuscation attacks. However, both models remained susceptible to more complex cross-modality attacks where harmful intent emerged only through multimodal interaction, indicating that advanced visual perception alone does not resolve cross-modal alignment deficits.

The researchers identified significant methodological flaws in existing safety evaluation paradigms, particularly the widespread reliance on binary refusal detection. Traditional evaluations calculate the adversarial success rate as one minus the refusal rate, categorizing any response lacking a predefined refusal string as an attack success. This binary approach inflates attack success rates by misclassifying incoherent, degraded, or evasive outputs as successful jailbreaks. The authors observed that safety alignment often degrades instruction-following capabilities, leading models to output nonsensical phrases such as addCriterion online polls instead of executing authentic safety-driven refusals.

To overcome these measurement limitations, the authors proposed a two-stage evaluation framework called ThreatsEval, implemented using DSPy and evaluated with GPT-4o-mini applying chain-of-thought prompting in triplicate with majority voting. The first stage introduces a three-way probability decomposition where the total response probability equals the sum of the refusal rate, the instruction non-following rate, and the true adversarial success rate. This formulation isolates capability degradation from true safety alignment, ensuring that model degradation is not falsely celebrated as alignment or conflated with compliance.

The second stage of the framework establishes fine-grained qualitative scoring of model behavior. Refusals are divided into hard refusals, which represent clear and explicit denials, and informative refusals, which decline the malicious task while offering constructive context, educational warnings, or ethical framing without enabling harm. To evaluate successful breaches, the framework introduces an adversarial quality score using a five-point Likert scale normalized between zero and one, assessing how specifically, clearly, and helpfully an output fulfills the harmful intent of an adversarial prompt.

Experimental evaluation across AdvBench, FigStep, MMSafetyBench, and SIUO benchmark datasets under fixed generation parameters revealed clear behavioral contrasts among the tested architectures. LLaMA-3.2-Vision exhibited an instruction non-following rate of seventy-six percent, frequently defaulting to generic privacy refusals due to overfitted identity-related safety filters. Consequently, it responded to only four percent of harmful prompts, resulting in low adversarial quality scores driven primarily by extreme conservatism and capability loss rather than nuanced contextual safety judgment.

Conversely, Qwen-2.5-VL and InternVL-2.5 demonstrated strong adherence to instructions but recorded higher adversarial success rates and quality scores on MMSafetyBench and SIUO datasets, where harmful concepts were split across modalities or concealed within ambiguous combinations. InternVL demonstrated the highest proportion of hard refusals at thirty-six percent, reflecting a conservative post-fusion alignment posture. Across all models, AdvBench produced the lowest adversarial success rates, showing that modern vision-language fusion architectures and post-fusion alignment have become increasingly resilient against simple gradient-based pixel perturbations.

The authors also highlighted the evaluative conflict surrounding professional advice categories, including legal, medical, and financial queries. While existing benchmark protocols demand outright refusal for professional queries, industrial implementations prefer informative responses paired with contextual warnings. When the researchers removed professional advice queries from benchmark scoring, measured adversarial success rates and quality scores dropped across the board, underscoring that benchmark design choices heavily shape perceived model safety.

Ultimately, the study underscores that robust multimodal safety requires moving beyond simplistic binary string matching and superficial guardrails. An idealized safety profile must achieve low adversarial success rates, minimal instruction degradation, and an appropriate balance between hard refusals for critical dangers and informative refusals for nuanced queries. By disentangling capability loss from genuine safety compliance and uncovering how visual projections encode latent risks, the research provides a systematic foundation for benchmarking and training secure multimodal foundation models.

Earlier efforts to evaluate model safety through automated judges highlighted significant gaps that informed the authors’ framework. Automated judging methods such as StrongReject attempt to measure model safety by evaluating how convincingly and specifically a model refuses harmful instructions, combining refusal, specificity, and convincingness scores. However, StrongReject primarily measures response quality rather than safety, creating vulnerabilities where a step-by-step description of an action, such as 3D printing a firearm, might receive high marks for clarity despite lacking actionable danger or conversely providing superficial compliance. To refine this, the ELITE benchmark incorporated a dedicated toxicity multiplier into the scoring formula, ensuring that outputs carrying explicit hazards are penalized while benign responses receive lower threat scores. Yet the authors observed that both StrongReject and ELITE still fail to account for model degradation or the complete loss of instruction-following capabilities, which misrepresents incoherent model outputs as successful adversarial breaches.

The study also places multimodal vulnerabilities within a broader trajectory of adversarial techniques that exploit architectural boundaries. Beyond basic typographic rendering, recent cross-modal jailbreaks employ increasingly structured visual formats, such as diagrams, flowcharts, or multi-image configurations that divide malicious prompts across independent inputs. In cross-modality separation frameworks like MIS, malicious phrases such as building an explosive device are dismantled into neutral individual components, placing action verbs and target nouns into separate visual or textual containers that appear harmless in isolation. Furthermore, hybrid attack protocols like HADES combine gradient-based pixel perturbations, visual encoding, and textual paraphrasing to maximize bypass success across multimodal interfaces.

These technical evaluations reveal significant interpretive complexities in defining multimodal harm, particularly when assessing subtle modality interactions. In benchmarks like SIUO, where individually benign text and imagery combine to produce unsafe interpretations, evaluating compliance requires navigating ambiguous cultural contexts. For example, pairing a neutral question about selecting an appropriate mother’s gift with an image of knitting yarn subtly touches upon gender stereotyping, presenting borderline scenarios where classifying a response as harmful remains inherently subjective. By openly releasing their evaluation pipeline and code under the SeeThreats repository, the researchers aim to establish standardized benchmarks that distinguish subtle cultural biases, genuine safety violations, and unintentional capability degradation across evolving vision-language architectures.

Subject of Research: Vulnerabilities of large vision-language models to multimodal adversarial attacks and evaluation methods

Article Title: Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack

Article References: Ren, J., Dras, M., & Naseem, U. (2026). Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack. Human-Centric Intelligent Systems, 6(2), 208-218. https://doi.org/10.1007/s44230-026-00141-w

Image Credits: AI Generated

DOI: 10.1007/s44230-026-00141-w

Keywords: vision-language models, multimodal safety, adversarial attacks, jailbreak vulnerabilities, representation analysis, ThreatsEval, adversarial success rate, instruction non-following, hard refusal, informative refusal, LLaVA, LLaMA-3.2-Vision, Qwen-2.5-VL, InternVL, FigStep, MMSafetyBench, SIUO, AdvBench

Cite Scienmag News

Blake Davidson. (November 19, 2026). Visual Inputs Expose Multimodal Vision-Language Models to Hidden Adversarial Vulnerabilities. Scienmag. https://scienmag.com/visual-inputs-expose-multimodal-vision-language-models-to-hidden-adversarial-vulnerabilities/

Blake Davidson. "Visual Inputs Expose Multimodal Vision-Language Models to Hidden Adversarial Vulnerabilities." Scienmag, 19 November 2026, https://scienmag.com/visual-inputs-expose-multimodal-vision-language-models-to-hidden-adversarial-vulnerabilities/. Accessed 19 November 2026.

Blake Davidson. "Visual Inputs Expose Multimodal Vision-Language Models to Hidden Adversarial Vulnerabilities." Scienmag. November 19, 2026. https://scienmag.com/visual-inputs-expose-multimodal-vision-language-models-to-hidden-adversarial-vulnerabilities/

Tags: AdvBenchadversarial attacksadversarial success rateFigStephard refusalinformative refusalinstruction non-followingInternVLjailbreak vulnerabilitiesLLaMA-3.2-VisionLLaVAMMSafetyBenchmultimodal safetyQwen-2.5-VLrepresentation analysisSIUOThreatsEvalvision-language models
Share26Tweet16
Previous Post

Preterm Infants Needing Eye Treatment Show Early Postnatal Weight Deficits

Next Post

Syphilis-Linked Strokes Frequently Strike Younger Patients via Inflammatory Arterial Obstruction

Related Posts

Congo Timber Study Decouples Sapelli Wood Spectral Variation From Genetic Structure
Science News

Congo Timber Study Decouples Sapelli Wood Spectral Variation From Genetic Structure

November 19, 2026
Study Identifies Silica Dust Exposure Threshold Linked to Accelerated Spirometric Decline
Medicine

Study Identifies Silica Dust Exposure Threshold Linked to Accelerated Spirometric Decline

November 19, 2026
Autonomous Drone Tracks Intruders in Proof of Concept for Solar Farms
Technology and Engineering

Autonomous Drone Tracks Intruders in Proof of Concept for Solar Farms

November 19, 2026
Clinical Omics Roadmap Outlines Precision Medicine Strategy for Eastern Mediterranean Health Systems
Medicine

Clinical Omics Roadmap Outlines Precision Medicine Strategy for Eastern Mediterranean Health Systems

November 19, 2026
Hybrid Neural Networks Improve Inverse Gompertz Gumbel Reliability and Failure Modeling
Science News

Hybrid Neural Networks Improve Inverse Gompertz Gumbel Reliability and Failure Modeling

November 19, 2026
Pharmacist Electronic Checklists Linked to Higher Quadruple Heart Failure Therapy Prescriptions
Medicine

Pharmacist Electronic Checklists Linked to Higher Quadruple Heart Failure Therapy Prescriptions

November 19, 2026
Next Post
Syphilis-Linked Strokes Frequently Strike Younger Patients via Inflammatory Arterial Obstruction

Syphilis-Linked Strokes Frequently Strike Younger Patients via Inflammatory Arterial Obstruction

  • Mothers who receive childcare support from maternal grandparents show more optimized

    Mothers who receive childcare support from maternal grandparents show more parental warmth, finds NTU Singapore study

    27656 shares
    Share 11059 Tweet 6912
  • University of Seville Breaks 120-Year-Old Mystery, Revises a Key Einstein Concept

    1061 shares
    Share 424 Tweet 265
  • Bee body mass, pathogens and local climate influence heat tolerance

    682 shares
    Share 273 Tweet 171
  • Researchers record first-ever images and data of a shark experiencing a boat strike

    546 shares
    Share 218 Tweet 137
  • Groundbreaking Clinical Trial Reveals Lubiprostone Enhances Kidney Function

    531 shares
    Share 212 Tweet 133
Science

Embark on a thrilling journey of discovery with Scienmag.com—your ultimate source for cutting-edge breakthroughs. Immerse yourself in a world where curiosity knows no limits and tomorrow’s possibilities become today’s reality!

RECENT NEWS

  • Congo Timber Study Decouples Sapelli Wood Spectral Variation From Genetic Structure
  • Study Identifies Silica Dust Exposure Threshold Linked to Accelerated Spirometric Decline
  • Autonomous Drone Tracks Intruders in Proof of Concept for Solar Farms
  • Clinical Omics Roadmap Outlines Precision Medicine Strategy for Eastern Mediterranean Health Systems

Categories

  • Agriculture
  • Anthropology
  • Archaeology
  • Athmospheric
  • Biology
  • Biotechnology
  • Blog
  • Bussines
  • Cancer
  • Chemistry
  • Climate
  • Earth Science
  • Editorial Policy
  • Marine
  • Mathematics
  • Medicine
  • Pediatry
  • Policy
  • Psychology & Psychiatry
  • Science Education
  • Science News
  • Social Science
  • Space
  • Technology and Engineering

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 5,151 other subscribers

© 2026 Scienmag - Science Magazine

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • HOME
  • SCIENCE NEWS
  • CONTACT US

© 2026 Scienmag - Science Magazine

Discover more from Science

Subscribe now to keep reading and get access to the full archive.

Continue reading