A new benchmarking framework is challenging a hidden weakness in artificial-intelligence research aimed at discovering which drugs might treat which diseases: many models are not being compared on equal terms. Called UIDDA, the system brings data preparation, feature extraction, machine-learning models and classification methods into a single, controlled pipeline, allowing researchers to test where predictive performance really comes from. In experiments across four public drug-disease association datasets, the choice of representation-learning model changed results far more than the choice of classifier. The finding could reshape how computational drug-repurposing studies are evaluated, because apparently impressive predictions can depend as much on preprocessing and experimental design as on the neural network at the center of the system.
Drug-disease association prediction is intended to identify therapeutic relationships that have not yet been confirmed experimentally. A drug may influence a molecular target, pathway or biological process involved in a disease, while a disease may share genetic, chemical or network-level similarities with conditions already treated by existing medicines. Computational systems attempt to combine these clues and rank unknown drug-disease pairs for laboratory testing. The attraction is clear: screening millions of possible relationships in silico can be faster and cheaper than testing every candidate in cells or animals. But the field has developed with widely varying datasets, definitions of negative examples, feature engineering procedures, model architectures and validation strategies, making results from different studies difficult to compare directly.
UIDDA, developed by researchers at Hainan Medical University in China, addresses this problem by separating the process into two major stages. First, a feature-extraction model converts drugs, diseases and their biological relationships into numerical representations. These representations can incorporate association matrices, molecular information, Gaussian interaction profile similarities and heterogeneous biomedical networks containing multiple entity and relation types. Second, a classification head estimates whether a particular drug-disease pair is likely to be associated. The framework tested nine feature-extraction models against four downstream classifiers, including multilayer perceptron, logistic regression, support vector machine and random forest approaches. By reusing the same inputs and evaluation structure, the researchers could measure whether a performance gain originated from the representation itself or from the final decision-making algorithm.
The distinction is technically important. A classifier receives a numerical description of a candidate pair and learns a boundary separating known associations from pairs treated as unknown or negative. A feature-extraction model, by contrast, determines what information is preserved before classification begins. In network-based systems, this may involve learning embeddings from graphs in which drugs, diseases, genes, targets and pathways are connected. In matrix-based approaches, the model may infer missing entries from patterns of similarity and interaction. If one study supplies a richer or more conveniently processed representation than another, the classifier may appear to be responsible for the improvement even when it is merely exploiting better input data. UIDDA’s unified-input design attempts to make that distinction visible.
The researchers also took aim at a major threat to biomedical machine learning: information leakage. Leakage occurs when information from the test set, directly or indirectly, influences the representation learned from the training set. In drug-disease prediction, the danger is especially subtle. Association matrices, Gaussian interaction profiles and network-derived features can all encode relationships involving test pairs. If those structures are constructed once using the complete dataset and then divided into training and test portions, the model may receive clues about the answers it is supposedly being tested on. UIDDA reconstructed association-dependent representations independently inside each training fold, ensuring that test information was withheld until evaluation. This procedure produces more demanding and more credible estimates of generalization.
Under the pair-level Random-U setting, in which drug-disease pairs were divided into training and test groups and unobserved pairs were handled as presumed negatives, two feature models delivered the strongest overall performance: FD-MSGL and AMDGT. DeepDR, HNetDNN and LDSCNN had lower average discrimination across the benchmark. The difference between the best and worst feature-extraction models was striking. Mean area under the receiver operating characteristic curve, or AUC, differed by 0.266 between the strongest and weakest models, whereas the corresponding gap among the four classification heads was only 0.032. AUC measures how effectively a model ranks positive associations above negatives across decision thresholds; a value of 0.5 is equivalent to random ranking, while higher values indicate better separation. The result suggests that choosing how biological information is represented is substantially more consequential than choosing the final classifier.
The benchmark also revealed that apparent performance depends heavily on how unknown associations are treated. In the Random-U evaluation, the overall AUC and area under the precision-recall curve, or AUPR, were 0.724 and 0.732, respectively. When the researchers used a Hard-U setting, designed to make unknown pairs more challenging, those values fell to 0.692 and 0.709. Under a non-negative-positive-unlabeled, or nnPU, treatment, they declined further to 0.675 and 0.694. The distinction matters because an unrecorded drug-disease relationship is not necessarily a true negative: it may simply have never been studied. Treating every unknown pair as negative can therefore reward models that recognize gaps in the database rather than genuine biology. Precision-recall analysis is particularly useful in this setting because drug-disease datasets are often highly imbalanced, with far fewer known associations than possible pairs.
Dataset composition created another layer of variation. The B-dataset produced the highest overall mean AUC, while the sparse T-dataset produced the lowest. Sparse association matrices contain fewer known links relative to the number of possible drug-disease combinations, leaving models with less evidence from which to learn reliable patterns. A model can perform well when related drugs, diseases or network neighborhoods are represented repeatedly in training, yet struggle when the benchmark contains little overlap or noisy annotations. This is why a single headline score can be misleading: it may reflect the density, curation history and biological scope of a dataset as much as the underlying algorithm. UIDDA’s cross-dataset comparison makes those differences part of the result rather than treating them as inconvenient variability.
The most demanding tests withheld entities rather than merely withholding pairs. In the drug-held-out setting, test drugs were excluded from training and the mean AUC was 0.692. When diseases were held out, mean AUC dropped to 0.614. When both test drugs and test diseases were unseen during training, the mean AUC reached only 0.552, barely above random ranking. These experiments probe a question closer to real drug discovery: can a model make useful predictions for a genuinely new compound, a newly characterized disease or a combination of both? Pair-level validation can be relatively forgiving because the same drug or disease may appear in both training and test sets through different pairings. Entity-held-out evaluation removes that familiarity and exposes how dependent a system is on recognizing previously encountered biological objects.
The team further tested whether adding MolVis-inspired three-dimensional molecular features would substantially improve FD-MSGL, one of the strongest models. The changes were small and depended on the dataset, suggesting that more detailed molecular geometry does not automatically translate into better association prediction when the dominant signal may lie in interaction networks or curated disease relationships. Finally, the researchers conducted a frozen top-10 retrospective literature assessment, checking whether highly ranked predictions had qualitative support in existing publications. That analysis provided a limited plausibility check rather than proof of clinical utility. UIDDA is a benchmarking framework, not a validated diagnostic or drug-development system, and its scores do not establish that any particular predicted association will work in patients. Its broader message is methodological: reliable progress will require leakage-controlled experiments, explicit treatment of unknown associations and tests that distinguish familiar entities from truly unseen ones. By making those factors visible, UIDDA could help turn a crowded race for higher scores into a more meaningful search for models that generalize to real biomedical discovery.
