Dual Modality Preference Modeling Boosts Accuracy in Multimodal Recommender Systems

Digital commerce platforms depend heavily on automated recommendation engines to guide customers through massive catalogs of merchandise, yet conventional recommendation systems often struggle when interaction histories remain scarce or incomplete. In a study published in Data Science and Engineering, researchers Shijie Zhu, Qian Zhou, Wei Chen, Shangfei Zheng, and Lei Zhao presented a framework titled HUMP, short for Highlighted Users Modality Preference. The system integrates diverse content modalities, specifically visual imagery and descriptive text, directly into the algorithmic recommendation architecture. Traditional collaborative filtering pipelines rely almost exclusively on past user-item interactions, leaving them vulnerable to cold-start obstacles whenever a newcomer registers or a novel inventory item enters the database. Although multimodal recommendation models have emerged to alleviate data sparsity by incorporating supplementary descriptive data, prior architectures frequently fail to fully leverage how individuals and communities interact with differing media types. The authors addressed these structural limitations by designing a dual-track preference fusion mechanism coupled with layered graph neural network convolutions, establishing an architecture that balances personal browsing inclinations against global product-level modality trends.

Multimodal recommender systems aim to extract richer latent embeddings by ingesting descriptive product features alongside historical behavioral feedback. Existing approaches generally fall into three structural categories: matrix factorization models that incorporate visual or textual features as auxiliary side signals, graph neural network models that propagate feature information across user-item bipartite graphs, and self-supervised architectures that devise contrastive learning objectives to optimize feature alignment. Despite these advancements, the authors identified two fundamental theoretical problems that curtail recommendation precision across contemporary models. First, existing data fusion strategies either ignore personalized multimodal inclinations entirely or focus strictly on isolated user preferences while overlooking the collective, item-centered global modality preference. Second, current models fail to glean deeper relational insights from user-item interactions once multimodal fusion takes place. Interaction graphs in commercial systems are inherently sparse, noisy, and skewed by long-tail consumption patterns, which leaves rarely viewed items with fragile neighborhood connections and uninformative representations. HUMP resolves these interrelated hurdles by explicitly modeling both individual and item-centered preference parameters, followed by structured message passing across an expanded heterogeneous graph.

To articulate the role of modality preferences, the researchers distinguished between an individual user’s consumption behavior and the broader behavioral profile of an item across all consumers. An individual user often exhibits an idiosyncratic bias toward specific forms of communication, allocating greater attention to either textual specifications or product photography. For instance, when evaluating an apparel item, a shopper might dedicate sixty-five percent of their evaluative focus to textual descriptions and material details while reserving thirty-five percent for visual photographs. Conversely, an item-centered global modality preference captures the average distribution of attention assigned to that specific product across the entire population of users. For certain items, community consensus might prioritize visual aesthetics, whereas other products demand extensive textual vetting regardless of individual shopping tendencies. Prior models like MMGCN, VBPR, and DualGNN either forced equal weighting across modalities or ignored how collective consensus can anchor users whose individual interaction histories are sparse. By synthesizing learnable individual preference parameters with item-centered global indicators, HUMP provides an adaptive mechanism that informs downstream representation aggregation.

The mathematical pipeline of HUMP operates through four sequential modules: single-modal representation learning, modality preference guided data fusion, global representation enhancement, and interaction probability prediction. In the initial single-modal stage, raw multimodal input data undergo modality-specific transformations through multilayer perceptrons equipped with LeakyReLU activation functions. This transformation projects arbitrary visual and textual feature vectors into a shared latent space. Structural connectivity is subsequently captured through graph convolutional operations built upon the LightGCN paradigm, which computes localized message passing over the user-item bipartite graph. To counter oversmoothing across propagating layers, the authors integrated residual connections directly into the message-passing formulations. Across L propagation layers, feature distributions from neighboring nodes are iteratively aggregated and summed together with the initial latent embeddings. This yields refined, modality-specific embedding vectors for every consumer and item across the textual and visual modalities, establishing the foundation for cross-modal integration.

Once single-modal representations are established, the preference-guided fusion module integrates these distinct vectors into unified user and item representations. Rather than applying uniform pooling or static concatenation, HUMP introduces trainable parameter vectors assigned to individual users and items, matching the total number of evaluated modalities. For each user, the model learns an individual modality preference vector, whereas each item is assigned a distinct item-centered global modality preference vector. The unified user embedding is formed by scaling the concatenated modality-specific representations by the user preference parameter, and the unified item representation is derived similarly by scaling concatenated item modalities with the global item preference weights. This formulation preserves the dimensional richness of the individual modalities while explicitly weighting their practical influence according to observed empirical interactions. By making these weights fully learnable during backpropagation, the model dynamically adapts to fine-grained preference distributions without requiring manually defined heuristic rules or costly per-user fusion networks.

To overcome graph sparsity and extract deeper behavioral patterns following multimodal fusion, HUMP executes a global representation enhancement phase. The researchers recognized that users with shared consumption habits tend to exhibit consistent preferences across single-modal and multimodal contexts alike, a principle that applies symmetrically to items. Drawing on top-K sampling strategies, the authors extracted co-occurrence matrices for both users and items based on historical interaction frequencies. By selecting the top-K most frequently co-occurring neighbors, the system constructs a user co-occurrence graph alongside an item co-occurrence graph. These derived structures are unified with the original user-item interaction edges to build a user-item layered heterogeneous graph. To systematically traverse this multi-relational network, the team designed two asymmetric metapaths: one linking users across shared item interactions and item co-occurrences to capture personalized modality inclinations, and another linking items through user interactions and user co-occurrences to preserve collective item modality profiles.

Message aggregation across these asymmetric metapaths is executed through a Heterogeneous Graph Attention Network, commonly known as HAN, which weighs the semantic importance of neighboring nodes and updates the latent representations through adaptive scaling coefficients. Following the attention mechanism, an additional graph convolutional network layer propagates signals globally across the entire layered heterogeneous graph, further reinforcing node embeddings through normalized neighbor averaging and residual connections. The resulting vectors represent the finalized, highly contextualized representations of users and items. In the terminal prediction module, the model estimates the interaction likelihood between a given user and a candidate item by taking the inner product of their finalized representation vectors. For parameter optimization, the framework employs the Bayesian Personalized Ranking loss function, constructing training triplets of one user, one observed consumed item, and one unconsumed negative sample, regularized through an L2 penalty term.

The authors evaluated HUMP on two real-world Amazon review benchmark datasets: Sports and Outdoors, and Baby. These datasets encompass diverse product catalogs containing both four-thousand-ninety-six-dimensional visual features extracted from product imagery and three-hundred-eighty-four-dimensional sentence embeddings derived from titles, descriptions, categories, and brands using sentence-transformer models. The raw interaction histories were filtered using a standard five-core iterative setting, ensuring every retained user and item possessed at least five recorded interactions, with review ratings treated as positive interaction signals. Interaction histories were divided using a random eight-to-one-to-one split across training, validation, and testing partitions. Feature inputs were standardized to a uniform sixty-four-dimensional input before being mapped into one-hundred-twenty-eight-dimensional latent embeddings.

Benchmarked against seven prominent baselines—spanning matrix factorization models such as VBPR, graph neural networks including MMGCN, DualGNN, and GRCN, and self-supervised learning architectures like BM3, SLMRec, and MMSSL—HUMP demonstrated consistent performance gains across all evaluated metrics. The authors tracked top-P ranking quality through Recall and Normalized Discounted Cumulative Gain at thresholds of five, ten, and twenty. The experimental records revealed that HUMP consistently surpassed competitive baselines, achieving a particularly notable improvement of eight point forty-seven percent in Normalized Discounted Cumulative Gain at threshold five on the Baby dataset. The researchers emphasized that HUMP outperformed self-supervised frameworks despite relying on a supervised Bayesian Personalized Ranking objective, indicating that structured modeling of dual-level modality preferences and layered graph propagation can compensate effectively for the absence of self-supervised auxiliary tasks.

Ablation experiments confirmed the synergistic nature of HUMP’s underlying design choices. Removing both individual and item-centered global preference parameters caused the steepest degradation in recommendation accuracy, underscoring that simple feature averaging across modalities discards essential signal. Eliminating either preference parameter independently also resulted in measurable performance declines, proving that personal inclinations and global item tendencies serve complementary roles. Removing graph convolutions from the layered heterogeneous structure caused significant drops in ranking efficacy, demonstrating the value of deep global propagation. Furthermore, stripping residual connections triggered substantial performance declines attributable to graph oversmoothing effects. Kernel density estimations across the Sports and Baby datasets revealed multimodal distributions with multiple distinct peaks for both individual and global preference values, visually confirming that users and items do not conform to uniform modality weights. In outlining prospective research trajectories, the authors highlighted opportunities to integrate Large Language Models into post-ranking phases, align external knowledge graphs to enhance semantic context, and refine propagation efficiency for hyperscale enterprise deployments.

Subject of Research: Multimodal recommender systems and graph neural network data fusion

Article Title: HUMP: Highlighted Users’ Modality Preference for Multi-modal Recommender Systems

Article References: Zhu, S., Zhou, Q., Chen, W., Zheng, S., & Zhao, L. (2025). HUMP: Highlighted Users’ Modality Preference for Multi-modal Recommender Systems. Data Science and Engineering, 10(4), 786-799. https://doi.org/10.1007/s41019-025-00309-7

Image Credits: AI Generated

DOI: 10.1007/s41019-025-00309-7

Keywords: multimodal recommendation, recommender systems, collaborative filtering, graph neural networks, LightGCN, heterogeneous graph attention network, data fusion, modality preference, Bayesian personalized ranking, e-commerce algorithms, representation learning, user modeling, Amazon review dataset, top-k sampling, cold-start mitigation, sentence embeddings, visual features