Gradient-Guided Sparse Modeling Reduces Task Interference in Multi-Task Learning

Researchers have developed GD-SMTL, a gradient-dependence-guided sparse multi-task learning framework designed to mitigate task interference and promote positive knowledge transfer across related computational tasks. Evaluated across diverse benchmarks, including multi-task image classification, multi-label face attribute prediction, and dense prediction, the method effectively suppressed interference during joint training.

Task interference represents a persistent challenge in multi-task learning, where optimizing several related tasks jointly within a single network can degrade individual performance. Existing mitigation techniques generally rely on sparse multi-task learning or optimization-based strategies. Sparse methods generate task-specific masks by independently estimating parameter importance to divide the shared backbone, but this strategy ignores cross-task parameter dependencies and dynamic gradient conflicts, leading to suboptimal mask allocation. Meanwhile, optimization-based approaches seek compromises among conflicting task gradients, which frequently causes suboptimal solutions for individual tasks and constrains overall multi-task performance.

To address the structural limitations of both approaches, the GD-SMTL architecture jointly incorporates parameter utility and cross-task gradient interactions into its sparse parameter allocation process. The framework calculates task-specific parameter utility through robustly normalized Fisher information. Concurrently, it captures persistent cross-task gradient interactions by tracking exponential moving average gradient trajectories during optimization.

Based on these captured interactions, the framework applies an asymmetric dependency-aware allocation strategy that assigns contested parameters according to relative task dependence. This approach enables targeted parameter distribution across shared layers while curbing interference between competing objectives. Experimental results across image classification, face attribute prediction, and dense prediction confirm that the gradient-guided framework curtails task conflicts while facilitating constructive knowledge sharing across tasks.

Subject of Research: Multi-task learning optimization

Article Title: Mitigating task interference in multi-task learning via gradient-guided sparse modeling

Article References: Zhang, Y., Li, J., Qian, Y., Wei, Z., & Zuo, J. (2026). Mitigating task interference in multi-task learning via gradient-guided sparse modeling. International Journal of Machine Learning and Cybernetics, 17(11), Article 525. https://doi.org/10.1007/s13042-026-03366-0

Image Credits: AI Generated

DOI: 10.1007/s13042-026-03366-0

Keywords: multi-task learning, task interference, sparse modeling, gradient conflicts, Fisher information, parameter allocation, deep learning, image classification, Mitigating, task, interference, multi-task