The review, which screened more than 400 candidate papers down to 120 rigorously assessed studies, arrives at a moment when the field is accelerating faster than most surveys can track. Earlier reviews, the authors argue, tended to focus narrowly on single domains such as medicine or language processing, leaving researchers without a comparative view of the architectures, datasets, and training objectives that define the current generation of models. By covering eleven application domains, including healthcare, education, autonomous transportation, e-commerce, and industrial manufacturing, the paper aims to give both newcomers and specialists a coherent picture of what these systems can actually do.

At the technical heart of the review lies a family of architectures built on the transformer, the attention-based design that revolutionized natural language processing before conquering vision. The Vision Transformer, or ViT, broke with convolutional tradition by slicing images into fixed 16-by-16-pixel patches, flattening them into vectors, and processing them exactly like words in a sentence. This simple reframing allowed vision models to inherit the scaling behavior of language models, and it became the visual backbone for much of what followed. CLIP, developed by OpenAI, went further by training an image encoder and a text encoder jointly on 400 million image-text pairs scraped from the internet, learning to pull matching pairs together in a shared latent space while pushing mismatched ones apart. That contrastive objective gave machines a rudimentary form of grounded understanding, and CLIP now serves as the visual front end for many multimodal large language models.

Subsequent generations refined this recipe in strikingly different directions. BLIP introduced a flexible encoder-decoder design and a bootstrapping mechanism called CapFilt, which generates synthetic captions and filters noisy training pairs to improve data quality. Its successor, BLIP-2, took a radically efficient path: rather than retraining everything, it froze both a pretrained image encoder and a large language model, connecting them through a lightweight Querying Transformer that translates visual features into a form the language model can consume. DeepMind’s Flamingo achieved few-shot multimodal learning by threading visual inputs into a frozen language model through a Perceiver Resampler and gated cross-attention layers, allowing it to answer visual questions from just a handful of examples. Google’s PaLI unified multilingual text generation via mT5 with high-capacity ViT encoders, while LLaVA pioneered visual instruction tuning, using GPT-4 to synthesize training dialogues and then aligning a CLIP vision encoder with a Vicuna language decoder.

The review also charts the frontier models that have captured public attention. GPT-4V extended OpenAI’s flagship language model to accept images, achieving human-level performance on many professional benchmarks while remaining weaker than humans on certain real-world tasks. GPT-4o, released in May 2024, went further still, processing text, images, audio, and video in a single network and becoming the first large language model to perform real-time emotion recognition from video. Florence-2, built on a Dual Attention Vision Transformer and trained on the FLD-5B dataset with 5.4 billion annotations across 126 million images, handles captioning, detection, segmentation, and grounding through one prompt-based sequence-to-sequence framework. SigLIP-2 replaced the conventional softmax with a sigmoid loss, decoupling batch size from training efficacy and supporting batches of up to one million examples while maintaining strong performance in more than 100 languages.

Efficiency has emerged as the defining engineering challenge, and the review documents how the newest models answer it. PaliGemma 2 pairs a SigLIP vision encoder with Gemma language models in sizes from 3 billion to 28 billion parameters, trained in three stages that progressively raise image resolution. Gemma 3 interleaves a single global attention layer with every five local layers using a restricted 1024-token sliding window, taming the memory demands of its 128,000-token context. Llama 4 adopts a sparse Mixture-of-Experts design in which a routing mechanism activates only a fraction of expert networks per token, expanding effective capacity while keeping inference cheap, and stretches context to roughly 10 million tokens for document-scale reasoning. These techniques, alongside parameter-efficient fine-tuning methods such as Low-Rank Adaptation, form what the authors call a crucial pathway to deployable multimodal AI.

The application evidence assembled in the review is striking in its breadth. In healthcare, fine-tuned combinations of vision transformers and language models reach up to 98.5 percent accuracy on lung disease diagnosis, while the PaliGemma-CXR system interprets tuberculosis chest X-rays with 90.32 percent accuracy. ChatIOS, which couples 3D point-cloud encoders with GPT-4V, achieved 93 percent intersection-over-union in automatic tooth segmentation from intraoral scans. Yet the same section catalogues sobering failures: GPT-4V identified anatomy correctly in 87.1 percent of radiology cases but pathology in only 35.2 percent, and diagnostic hallucination rates exceeding 40 percent were reported in some evaluations. In transportation, a fine-tuned PaliGemma model read license plates with 97.66 percent character accuracy, while GPT-4o managed 67 percent on pedestrian behavior prediction. In manufacturing, a CLIP-based defect classifier hit 99.9 percent AUROC with 6.6-millisecond inference, fast enough for real-time production lines.

Against these successes, the review is candid about the field’s structural weaknesses. Data remains the first bottleneck: many studies rely on small, single-center, or single-vendor datasets, annotations from lone experts, and corpora so noisy that models learn spurious correlations, the authors’ memorable example being polar bears on ice. Computational cost is the second, with models scaling to 120 billion parameters and demanding specialized hardware that restricts access for smaller research groups. Reliability is the third: models hallucinate plausible but incorrect findings, struggle with composite figures and micro-expressions, misclassify small pedestrians at low pixel density, and remain vulnerable to adversarial attacks such as projected gradient descent. Ethical concerns compound the technical ones, since multimodal datasets encode societal biases that models can perpetuate in sensitive domains like healthcare and education, and training on medical images raises privacy stakes that demand techniques such as federated learning and differential privacy.

The authors close with a research agenda that reads as a diagnosis of the field’s growing pains. They call for multi-center, human-annotated datasets that pair imaging with genetic and clinical data; for encoder refinements that can handle long audio and raw high-resolution images directly; for larger but memory-efficient transformers; and for explainability to be treated as a core evaluation requirement rather than an afterthought. Techniques like model distillation, which compresses large multimodal systems into deployable smaller versions, and Mixture-of-Experts scaling are highlighted as the most promising routes to accessible systems. What emerges from the 120 studies is a field in transition: the architectural foundations are largely settled, the benchmarks are impressive, but trust, efficiency, and generalization to the messy diversity of real-world data remain the unfinished work. For researchers deciding where to invest the next five years, this review functions as both a map and a warning.

Subject of Research: Pretrained multimodal deep learning models, their architectures, applications, and future research directions

Article Title: A comprehensive review on recent pretrained multimodal deep learning models from architectures to future directions

Article References: Hadi, A. A., & Supreethi, K. P. (2026). A comprehensive review on recent pretrained multimodal deep learning models from architectures to future directions. Discover Informatics, 1(1), Article 21. https://doi.org/10.1007/s44564-026-00022-1

Image Credits: AI Generated

DOI: 10.1007/s44564-026-00022-1

Keywords: multimodal deep learning, CLIP, vision transformers, GPT-4o, BLIP, Flamingo, PaliGemma, Llama 4, vision-language models, healthcare AI, Mixture-of-Experts, model hallucination

The research was carried out at the Affiliated Eye Hospital of Jiangxi Medical College, Nanchang University, in China, and rests on one of the largest datasets ever assembled for this purpose. The retrospective analysis encompassed 103,237 ophthalmic ultrasound images and 51,618 diagnostic reports drawn from 51,618 patients. The scale matters. Teaching an artificial intelligence system to read an ultrasound image and describe it the way an experienced ophthalmologist would requires tens of thousands of paired examples of images and the corresponding clinical text. The ethics committee of the hospital approved the study, and because it was a retrospective analysis of de-identified existing data, the requirement for individual informed consent was waived.

At the technical heart of OphthUS-GPT is a two-stage framework built on the Bootstrapping Language-Image Pre-training model, widely known as BLIP, a vision-language architecture that learns to connect visual content with descriptive text. The researchers fine-tuned BLIP in two distinct stages. In the first stage, the model was trained to produce clinical findings—the objective descriptions of what appears in the ultrasound image, such as the presence of echogenic material in the vitreous cavity or an elevated retinal membrane. In the second stage, the model learned to integrate those findings with clinical impressions, producing the complete diagnostic report structure that ophthalmologists use in daily practice. This staged approach mirrors the way human clinicians actually reason: first observing, then interpreting.

But generating a report is only half the problem. A written description of an ultrasound scan is of limited value to a patient, a trainee, or even a non-specialist physician if it cannot be understood. To address this, the team incorporated a large language model—DeepSeek, deployed locally—into the system to serve as an interactive question-answering module. Users can ask follow-up questions about a generated report, request clarification of medical terminology, or explore the clinical implications of specific findings, and the system responds through multi-turn dialogue. The result is not merely an automated typist but an interpretive assistant designed to make ophthalmic ultrasound findings accessible and actionable.

Evaluating such a system demands rigorous, multi-dimensional testing, and the researchers deployed three complementary strategies. First, they measured text quality using standard natural language generation metrics: BLEU, which measures n-gram overlap with reference reports; ROUGE, which captures longer matching sequences; and CIDEr, which weighs consensus with human-written descriptions. OphthUS-GPT achieved a BLEU-1 score of 0.5739, a ROUGE-L score of 0.6131, and a CIDEr score of 0.9818 for report generation—figures indicating substantial agreement between machine-generated text and the reports written by clinicians.

Second, the team assessed diagnostic performance through disease classification metrics, measuring accuracy, sensitivity, specificity, and F1 score across the range of conditions that appear in posterior segment ultrasound. The results were strongest for common conditions. For the majority of frequently encountered findings—including vitreous opacities, retinal detachment, posterior scleral staphyloma, and cataracts—classification accuracy exceeded 90 percent. Across all evaluated disease categories, overall accuracy exceeded 80 percent, a level of performance that suggests the system could function reliably as a first-pass drafting and screening tool even for less common pathology.

Third, and perhaps most importantly for clinical credibility, expert ophthalmologists rated the generated reports for accuracy and completeness. More than 90 percent of the reports produced by the system received scores of 3 or higher—the evaluation scale’s threshold for acceptable—on both criteria. This expert-validated quality is crucial, because statistical similarity metrics alone cannot guarantee that a report is medically sound. A report can match the vocabulary of its references while missing a critical finding; human review remains the gold standard, and by this measure OphthUS-GPT performed well.

The interactive question-answering module was evaluated separately, on four dimensions: accuracy, completeness, security, and user satisfaction. Here the study produced one of its most interesting comparative findings. The locally deployed DeepSeek model demonstrated statistically superior accuracy compared with Claude, and higher user satisfaction compared with both Claude and GPT-4 Turbo. No significant differences emerged between the models in completeness or security. The comparison matters for real-world deployment: large language models differ in how faithfully they answer medical questions and how satisfied clinicians and patients are with their responses, and the finding suggests that a locally hosted open model can outperform commercial alternatives on the dimensions that matter most, while also addressing data privacy concerns that often constrain hospital AI adoption.

The implications extend well beyond a single hospital in Nanchang. Manual ultrasound report generation is time-consuming and heavily dependent on clinician expertise, and in many parts of the world—particularly in lower-resource settings and in clinics without an on-site ultrasound specialist—ophthalmic B-scan interpretation is a genuine bottleneck in eye care. An automated system that produces expert-quality reports and explains them conversationally could shorten reporting times, standardize documentation quality, support less experienced examiners, and free clinicians to spend more time with patients and less time at the keyboard. Because the system includes an interpretive dialogue layer, it could also serve as a teaching tool for ophthalmology trainees learning to read ultrasound images, offering immediate, interactive explanations of findings and terminology.

The work, led by Fan Gan, Lei Chen, Weiguo Qin, and colleagues including corresponding author Zhipeng You, with Fan Gan and Lei Chen contributing equally, was supported by the Jiangxi Provincial Health Commission and the Public Hospital High-Quality Development Research Public Welfare Project. It represents a broader shift in medical artificial intelligence: away from narrow, single-task classifiers and toward integrated multimodal systems that combine computer vision, natural language generation, and conversational reasoning in a single clinical workflow. Challenges remain before such systems become routine—the evaluation was retrospective, prospective clinical deployment would require further validation across diverse populations and equipment, and any AI-generated report would still need clinician review. But the study demonstrates that the pieces can fit together. A vision-language model trained on more than one hundred thousand real ultrasound images can produce reports that experts judge accurate and complete, and a large language model layered on top can turn those reports into an interactive clinical conversation. For a diagnostic field where the written report has long lagged behind the image itself, that is a meaningful step forward.

Subject of Research: A multimodal artificial intelligence system for generating and interpreting ophthalmic B-scan ultrasound diagnostic reports

Article Title: OphthUS-GPT: a multimodal AI system for ophthalmic B-scan ultrasound report generation and interpretation

Article References: OphthUS-GPT: a multimodal AI system for ophthalmic B-scan ultrasound report generation and interpretation. (n.d.). https://doi.org/10.1186/s40537-026-01563-w

Image Credits: AI Generated

DOI: 10.1186/s40537-026-01563-w

Keywords: artificial intelligence, multimodal learning, ophthalmic B-scan ultrasound, report generation, BLIP, DeepSeek, large language models, ophthalmology, medical imaging, clinical decision support, vision-language models, deep learning