New LUISA algorithm leverages causal relationships for smarter feature selection

Machine learning models are only as good as the features they are built upon, yet many of the algorithms used to pick those features quietly mistake coincidence for cause. A team of researchers at the Pontifical Catholic University of Minas Gerais in Brazil has now proposed a new way to address that problem. In a study published in the International Journal of Data Science and Analytics, Walisson Ferreira de Carvalho, Beethoven M. Andrade, and Luis Enrique Zárate introduce LUISA, an algorithm that selects features by tracing causal relationships in data rather than relying on simple statistical association, and that deliberately looks beyond the narrow set of variables most existing causal methods consider relevant.

The core problem the researchers set out to solve lies in how feature selection is traditionally performed. Conventional filter and wrapper methods evaluate features using a single criterion, most often correlation with the target variable or improvement in classification accuracy. That approach can select features that are merely spurious correlations or direct consequences of the class rather than its causes. Causality-based methods, which represent relationships among variables in a causal Bayesian network, are more principled, but they too carry a hidden limitation: nearly all of them restrict relevant features to the Markov blanket of the target variable, a concept formalized by Judea Pearl. The Markov blanket comprises a target’s direct parents, its direct children, and the other parents of those children, known as spouses. Statistically, it is the minimal set of variables that renders the target conditionally independent of everything else. But as the Brazilian team points out, confining relevance to that blanket can discard features that genuinely matter.

The theoretical motivation for expanding the blanket comes from work on feature relevance classification, which divides attributes into strongly relevant, weakly relevant, and irrelevant categories. Weakly relevant features, such as the indirect ancestors of a target, can become strongly relevant in certain contexts, particularly through phenomena like Simpson’s paradox, in which a dependence between two variables disappears once a third variable is taken into account. Siblings of the target, variables sharing a common parent with it, can also shift in relevance and may even transform strongly relevant blanket members into weak ones. A direct parent, moreover, may itself be a feature computed from an indirect ancestor, in which case the more distant cause is the more informative variable. LUISA was designed precisely to capture this richer neighborhood of relevance.

Mechanically, LUISA operates in four stages and follows a divide-and-conquer strategy built on local rather than global learning of the Bayesian network. First, it identifies the direct parents and children of the target using the max-min parents and children algorithm, considered state of the art for this task, which relies on conditional independence tests using the G-squared statistic with a significance level of 0.05. Second, it discovers the parents of those parents and the children of those children, thereby extending the search to indirect ancestors and descendants and, in the process, capturing spouses and siblings of the target. Third, it performs a backward elimination phase in which candidate features are evaluated using partial correlation, a measure of the association between two variables after removing the influence of all others. If the partial correlation between a candidate and the target falls below a threshold of 0.2, the feature is discarded as causally irrelevant. Finally, the algorithm constructs a directed acyclic graph over the surviving features, orienting edges according to partial correlation, which serves as a proxy for direct causal effect and helps resolve the ambiguity that Markov’s observational equivalence creates when chains and forks produce identical patterns of conditional dependence.

The researchers evaluated LUISA from five perspectives, beginning with synthetic datasets in which the true causal features are known by construction. On the corral dataset, LUISA achieved a 100 percent success rate on the standard success indicator, which weighs selected relevant features against selected irrelevant ones. Notably, the algorithm recovered attribute 3, a genuinely relevant feature that classical Markov blanket methods such as MMHC and the PC algorithm miss entirely, because that feature lies outside the blanket. It also flagged attribute 6, a highly correlated but non-causal feature, as a candidate the authors suggest removing through post-processing. On the harder corral augmented dataset, which adds 93 noisy attributes, LUISA again scored 100 percent, surpassing the previous best result from the minimum redundancy maximum relevance method. On the monk3 dataset, it selected attributes 2 and 5, an efficient subset that together generates one conjunction of the class rule, achieving 67 percent, a figure exceeded only by ReliefF and an SVM wrapper.

Benchmark comparisons against twelve other selection methods, including causal and non-causal filters, wrappers, and NOTEARS-M, a score-based global causal structure learner extended for mixed data types, placed LUISA at the top overall, with an average success rate of 89.0 percent against 83.3 percent for the runner-up, ReliefF. Intriguingly, NOTEARS-M fell into exactly the trap LUISA was built to avoid: on the corral datasets it selected only attribute 6, the correlated non-causal feature, as a parent of the target, illustrating that even sophisticated global structure learning can mistake correlation for generation. Execution times remained modest, ranging from 0.24 seconds on corral to 6.75 seconds on monk3, reflecting the computational advantage of local learning, whose asymptotic complexity is dominated by the max-min parents and children procedure rather than by the NP-hard problem of learning an entire network.

The team then turned to five real-world datasets, testing whether models trained on LUISA’s reduced feature sets could match or beat those trained on the complete data. Classification was performed with decision trees, random forests, logistic regression, and naive Bayes, under a rigorous protocol including a 70/30 train-test split, undersampling to handle class imbalance, tenfold cross-validation, and five percent significance confidence intervals, with performance judged on accuracy, precision, recall, F1 score, and the Matthews correlation coefficient. On the heart disease dataset, LUISA selected just three of thirteen features, and the resulting models outperformed those built from all variables, with the decision tree reaching 84.72 percent accuracy and a Matthews coefficient of 0.695. On hepatitis, three of twenty features sufficed, and every classifier improved, with the random forest hitting 80 percent accuracy and a Matthews coefficient of 0.612. On the divorce prediction dataset, three features out of fifty-five reproduced the full dataset’s performance almost exactly, at 98.04 percent accuracy, while the random forest’s F1 confidence interval stretched from 90.3 to 100 percent.

The most striking result came from a sepsis dataset containing gene expression biomarkers from just 70 patients but 8,520 features, an extreme high-dimensional, low-sample regime that typically cripples machine learning and invites overfitting. LUISA selected six features, among them attribute 8420, a recognized biomarker directly associated with the etiology of sepsis. Critically, that biomarker did not appear among the top features chosen by conventional learning models trained on the full dataset, underscoring the algorithm’s ability to surface biologically meaningful signals beyond the statistical boundaries of the Markov blanket. A decision tree built on LUISA’s six features achieved 85.71 percent accuracy and a Matthews coefficient of 0.761, compared with 71.43 percent accuracy and 0.427 for the same classifier on the complete data, a dramatic improvement attributable to the removal of noise in a dataset where sample size widens confidence intervals considerably.

Against non-causal filters such as forward selection, backward elimination, and genetic-algorithm-based methods, LUISA delivered the best random forest F1 scores on the hepatitis and divorce datasets while using far fewer features, and even where it was narrowly outperformed in raw accuracy, its Matthews correlation coefficients were consistently higher, indicating more reliable models. Compared with the causal MMPC algorithm, LUISA selected strictly smaller feature sets, containing most or all of MMPC’s picks, and achieved higher Matthews coefficients on three of the four datasets, winning on three quarters of them overall. The authors emphasize that traditional filters, while competitive on accuracy, offer no guarantee of representativeness for the problem domain, since they can select effect features with high correlation to the class rather than true causes, undermining the explanatory value of the resulting model.

The study is not without acknowledged limitations. The 0.2 threshold for partial correlation was set from prior experimental work and may need domain-specific adjustment, since higher thresholds risk selecting variables that are merely linearly related to the class. The authors propose future refinements including LASSO regularization to prune spurious edges, validation of selected features through total causal effect analysis, and extension of the approach to other machine learning tasks. They also suggest incorporating prior domain knowledge to constrain the search. Even so, the results position LUISA as a notable step toward feature selection that yields not just accurate models but interpretable, causally grounded ones, able to distinguish genuine drivers from statistical echoes and to reveal knowledge that conventional pipelines would never surface.