Hybrid AI Model Blends Transformer and BiLSTM to Predict Cancer Drug Synergy

One of the most stubborn bottlenecks in modern oncology is not finding new drugs, but finding the right pairs of existing drugs that work better together than either does alone. Combination therapy can amplify treatment efficacy, slow the emergence of resistance, and reduce systemic toxicity, yet the number of possible drug pairings across thousands of compounds and hundreds of cancer cell lines grows so quickly that laboratory screening cannot keep pace. In vitro synergy assays remain slow, expensive, and labor-intensive, leaving most of the chemical space of possible combinations unexplored. A new study published in Discover Artificial Intelligence by Sahar Abbasi Rostami and Amir Lakizadeh of the University of Qom in Iran addresses this gap with a hybrid deep learning architecture called BT-Synergy, which the researchers report achieved an accuracy of 0.8458 on benchmark datasets for classifying synergistic drug combinations.

The central problem that BT-Synergy tackles is representational. Earlier computational approaches to synergy prediction, including AuDNNsynergy, SynPathy, and the widely used DeepSynergy model, relied on engineered molecular descriptors or structured multi-omics inputs. More recent frameworks such as SynergyX, DFFNDDS, SYNPRED, PRODeepSyn, DeepTraSynergy, and CFSSynergy jointly encode drug structures and cell-line characteristics using attention mechanisms, feature fusion modules, or protein-protein interaction networks. Yet many of these methods depend on SMILES string encodings or protein similarity matrices, which can miss higher-order chemical and biological dependencies. The Qom team argues that what is needed is an encoder that simultaneously understands the sequential grammar of a molecule and the long-range contextual relationships distributed across it, joined with a biologically grounded picture of the cell in which the interaction takes place.

To build that encoder, the researchers turned to SELFIES, a self-referencing molecular string representation that guarantees chemically valid outputs, unlike SMILES, which can produce structurally impossible sequences that corrupt downstream learning. Each drug is tokenized into SELFIES subunits, truncated or padded to a fixed length, and then processed by a hybrid module in which a bidirectional long short-term memory network is integrated directly into a Transformer block. In the final configuration, the BiLSTM actually replaces the conventional feed-forward sublayer inside the Transformer encoder. This design choice is deliberate: the Transformer’s multi-head self-attention excels at capturing long-range, non-local dependencies across a molecular sequence, while the BiLSTM contributes sequential inductive biases, reading the token stream in both forward and backward directions to preserve local structural patterns that attention alone can dilute.

The architecture was not chosen blindly. The team systematically compared variants, including a GRU-Transformer hybrid, a parallel configuration in which BiLSTM and Transformer pathways process the sequence simultaneously before element-wise fusion and layer normalization, and pure BiLSTM or pure Transformer baselines. Model depth also mattered: reducing the Transformer to two layers slightly degraded accuracy, while four or five layers inflated computational cost without commensurate gains. Three layers emerged as the optimal balance and were adopted in the final model. Ablation experiments confirmed that the hybrid design outperformed each single-architecture variant under identical conditions, supporting the premise that global contextual modeling and sequential dependency learning are complementary rather than redundant.

Equally important is how BT-Synergy represents the cellular context. Rather than relying on manually curated similarity networks, the model constructs cell-line embeddings from pre-trained protein language models. For each drug-cell-line instance, the researchers compile the union of proteins that are either annotated drug targets or observed as expressed in the relevant cancer cell line, drawing on drug-protein interaction and cell-line protein expression matrices inherited from the DeepTraSynergy dataset. This union typically spans between 54 and 1,479 proteins per sample, averaging roughly 354. Each protein’s canonical amino acid sequence is retrieved from UniProt and encoded with ProteinBERT from the TAPE suite, which produces dense vectors capturing local residue motifs and longer-range sequence dependencies. A learnable attention-based pooling layer then weights each protein embedding by its relevance, aggregating them into a single fixed-size cell representation that can be trained end-to-end with the rest of the network.

Fusion of the chemical and biological streams happens through a dual-fusion module designed to capture higher-order cross-modal interactions. The two drug embeddings are concatenated, then combined with the cell-line embedding via element-wise multiplication, addition, and subtraction. Multiplication emphasizes synergistic effects, addition captures complementary relationships, and subtraction highlights contrastive signals between molecular and cellular modalities. This interaction-aware scheme replaces naive concatenation, which a baseline variant confirmed is less effective. Because drug combinations are biologically symmetric, the team also applied order-invariance augmentation, generating mirrored training samples in which the two drugs are swapped. The augmentation paid off: across five cross-validation folds, predictions for original and reversed pairs showed a correlation of 0.9721 with a mean absolute difference of just 0.0511, indicating the model treats drug order symmetrically as biology demands.

Training and evaluation relied on two heterogeneous benchmarks. DrugCombDB contributed 69,436 drug-pair-cell-line observations spanning 764 compounds and 76 cancer cell lines, scored with the Zero Interaction Potency metric, whose values cluster tightly around zero. OncologyScreen, by contrast, contains 4,176 observations across 29 compounds and 21 cell lines, scored with the Loewe additivity model, which spans a far wider numerical range. To harmonize these divergent scales and combat class imbalance, the researchers adopted a quantile-based discretization: pairs in the upper quartile of each dataset’s score distribution were labeled synergistic, those in the lower quartile non-synergistic, and the ambiguous middle half was excluded. A sensitivity analysis comparing 50/50, 33/67, and 25/75 thresholds showed that including low-confidence pairs introduces substantial label noise. The strictest 25/75 configuration delivered the best trade-off, with accuracy of 0.8458, AUC-ROC of 0.9229, and F1 of 0.8422 on DrugCombDB.

The model also held up under punishing robustness protocols. In leave-one-drug-out evaluation, where all combinations involving held-out drugs are removed from training, BT-Synergy achieved an AUC-ROC of 0.8349; in leave-one-cell-line-out testing it reached 0.8357, suggesting genuine resilience to unseen drugs and biological contexts. When trained exclusively on DrugCombDB and tested on the entirely non-overlapping OncologyScreen dataset, the model retained encouraging predictive performance, providing preliminary evidence of cross-dataset transfer, though the authors caution that differing synergy-scoring systems limit strong generalizability claims. Interpretability analyses reinforced the picture: attention heatmaps revealed both globally distributed attention, integrating distant structural components, and sharply localized focus on chemically salient SELFIES symbols such as branching indicators, double-bond notations, and heteroatom tokens. In a token ablation experiment, masking the highest-attention fragments dropped one predicted synergy probability from 0.476 to 0.175, a striking decrease that suggests the model’s decisions hinge on specific molecular motifs, although the researchers stress that attention weights are proxy indicators rather than proven mechanisms.

Per-drug subgroup analysis added a biologically coherent note. Among the 29 OncologyScreen compounds, the model performed best on drugs with well-characterized mechanisms of action: 5-fluorouracil, an antimetabolite targeting thymidylate synthase, achieved a per-drug AUC-ROC of 0.9354, methotrexate, which inhibits dihydrofolate reductase, scored 0.9055, and doxorubicin, a DNA-targeting agent, reached 0.8850. Compounds with broad, pleiotropic, or poorly defined pharmacology fared noticeably worse. Across all 21 cancer cell lines, performance remained stable, with AUC-ROC values generally between 0.72 and 0.85, indicating the protein-informed cell representations prevent over-specialization to particular cellular backgrounds. Compared against DeepSynergy, GraphSynergy, NEXGB, DeepTraSynergy, and CFSSynergy, BT-Synergy delivered competitive performance on both benchmarks, an outcome the authors attribute to the combination of chemically valid SELFIES encoding, the BiLSTM-Transformer hybrid, and biologically informed protein embeddings.

The limitations are candidly acknowledged. Quantile-based binarization excludes half of the experimental spectrum, so reported performance reflects clearly defined observations rather than the full continuous distribution of synergy. Differences between ZIP and Loewe scoring constrain interpretations of transfer learning, and data sparsity plus the multi-target nature of complex biology mean performance will vary across contexts. The authors call for future validation using harmonized synergy measurements, continuous-label prediction, and additional independent pharmacological benchmarks, alongside extensions to multi-drug combinations and richer omics modalities. Even with those caveats, BT-Synergy demonstrates that fusing sequence-aware molecular encoders with protein language model embeddings can push drug synergy prediction toward the accuracy and robustness that precision oncology demands, and with the source code released on GitHub and both datasets publicly available, the framework is positioned to be tested, extended, and potentially deployed in the search for the next life-extending drug combination.

Subject of Research: A hybrid deep learning model combining BiLSTM and Transformer architectures with protein embeddings to predict synergistic cancer drug combinations

Article Title: A hybrid BiLSTM transformer model for drug synergy prediction

Article References: Rostami, S. A., & Lakizadeh, A. (2026). A hybrid BiLSTM transformer model for drug synergy prediction. Discover Artificial Intelligence, 6(1), Article 1182. https://doi.org/10.1007/s44163-026-02262-4

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02262-4

Keywords: drug synergy prediction, BT-Synergy, BiLSTM, Transformer, SELFIES, ProteinBERT, combination therapy, cancer cell lines, DrugCombDB, deep learning, computational pharmacology, protein embeddings