New Boosting Method Learns Oblique Rules for Simpler, More Interpretable Models

Interpretable machine learning has long faced a stubborn trade-off: models simple enough for humans to understand often sacrifice accuracy, while highly accurate models become inscrutable thickets of thousands of rules. A team of researchers at Monash University and the University of Haifa now reports a way to loosen that trade-off at its mathematical root. In a study published in Data Mining and Knowledge Discovery, Shahrzad Behzadimanesh, Pierre Le Bodic, Geoffrey I. Webb, and Mario Boley introduce a gradient boosting method called LLTBoost, which learns rules built on sparse linear combinations of input variables rather than single-variable thresholds. Across 14 benchmark regression and classification tasks, the resulting ensembles achieved lower complexity than competitive baselines while maintaining similar or better predictive accuracy, suggesting that interpretability and performance need not be enemies.

The starting point of the work is the additive rule ensemble, a prediction model composed of a small set of if-then rules whose outputs are summed, each weighted by a coefficient. In the classical formulation, a rule’s condition is a conjunction of simple threshold propositions such as a single variable exceeding a value, for example age greater than 45 or body mass index above 30. Geometrically, each such rule carves an axis-parallel region out of the input space, a rectangle or box whose faces align with the coordinate axes. This structure is what makes the models so readable: a human can mentally evaluate each threshold comparison, combine them with logical conjunctions, and trace exactly why a prediction was made. The authors note that these properties, known as simulatability and modularity in the explainable AI literature, are precisely what large ensembles like random forests lose, since even though forests can be expressed as rule ensembles, their sheer scale defeats human comprehension.

The catch is that axis-parallel rules are only compact when the input features are well curated. If the true decision boundary of a problem runs diagonally through the data, a box-shaped rule cannot capture it in one stroke; instead, the ensemble must stack many small axis-parallel regions in a staircase pattern, inflating both the number of rules and their length. The Monash team’s solution is to generalize the atomic proposition from a single-variable threshold to a weighted linear inequality, where a learnable, sparse weight vector combines several input variables before comparison with a threshold. Geometrically, this turns the decision regions from axis-parallel boxes into general polyhedra with oblique, slanted faces. Crucially, the weights are kept sparse, meaning only a handful of variables contribute to any single inequality, so each proposition remains something a person can plausibly compute in their head.

The authors illustrate the payoff with a diabetes risk model built from the 2017-18 CDC NHANES survey. A conventional axis-parallel rule ensemble and their oblique version achieved approximately identical log loss, but the oblique model did so with a markedly simpler rule set. The key was that, combined with log transforms, the learnable linear conditions allowed the model to represent ratio concepts automatically, in this case effectively reconstructing body mass index from height and weight, a feature that would otherwise have to be supplied by manual feature engineering. This is the deeper significance of the method: it equips rule ensembles with a linear, and therefore still principally interpretable, form of representation learning, letting the model discover interaction features like products and ratios on its own rather than depending on a domain expert to hand them over.

Making this idea algorithmically practical required solving a hard optimization problem inside every boosting round. The method builds on fully corrective gradient boosting, which starts from an empty model and iteratively adds the rule condition that maximizes a boosting objective, here the gradient sum, while re-optimizing all rule weights by convex optimization after each addition. The authors’ central technical insight is that finding the optimal oblique proposition for the gradient sum objective reduces to a weighted linear classification problem: the labels are the signs of the loss gradients over the training examples, the weights are the gradient magnitudes, and the goal is to find a sparse hyperplane that separates them. Because directly minimizing the 0/1 loss is intractable, they substitute an l1-regularized logistic loss, using a bisection search over the regularization parameter to land on a solution with exactly the desired number of non-zero weights, then refit the selected coefficients without regularization to remove estimation bias. The efficient LibLinear solver handles the underlying convex problems.

On top of this building block, the team designed a search strategy that avoids a subtle but important pitfall. A naive adaptation of greedy rule growing would always prefer oblique cuts over adding new simple propositions, encouraging dense linear transformations and wasting the complexity budget. Instead, their method systematically enumerates candidate rule conditions at every complexity level from one up to a maximum, allowing previously learned propositions to be refitted with higher sparsity as the budget grows, and then selects among the level-wise candidates using validation risk rather than the raw boosting objective. This ensures the search can discover combinations of genuinely simple propositions instead of prematurely committing to complicated ones.

The computational analysis shows the approach is surprisingly cheap. The total worst-case running time is linear in the number of training examples, in contrast to axis-parallel methods that rely on presorting data for fast cut-point search, and the bisection search over regularization values behaves essentially as a constant factor in practice, requiring fewer than 20 iterations across all benchmark datasets. For the small ensembles targeted by interpretable modeling, the cost of refitting rule weights is negligible relative to condition learning, so LLTBoost runs without substantial overhead compared to traditional axis-parallel rule boosting, a claim the timing experiments confirm directly.

The empirical evaluation compared LLTBoost against four families of competitors: traditional gradient boosting of axis-parallel rules, a RuleFit-style generate-and-select pipeline built on forests of oblique trees, neural rule ensembles that combine tree-extracted rules with continuous optimization, and explainable boosting machines that model the target as sums of univariate step functions plus pairwise interactions. Because every method can trade accuracy against complexity, the authors compared full risk-complexity Pareto fronts, measuring model complexity as the sum of rule counts, proposition counts, and non-zero weight entries, and normalizing test risk by the variance or entropy of an empty model so results are comparable across datasets. LLTBoost achieved the lowest overall test risk on the majority of the 14 datasets within a complexity budget of 100, and when the authors scanned each method’s Pareto front for the minimum complexity needed to hit risk-reduction targets of 25, 50, and 75 percent, LLTBoost required the least complexity on most datasets, with the advantage widening at the demanding 75 percent target.

The authors are candid about the limits of their complexity metric. Counting non-zero weights treats an oblique inequality as roughly equivalent to a simple threshold, but the cognitive effort of mentally evaluating a weighted sum of several variables is genuinely higher than checking whether one number exceeds another. Quantifying that difference, they note, is a question for interdisciplinary work with cognitive science. Still, they point to practical mitigations within machine learning itself, such as restricting weight values to small integer ranges in the style of integer scoring systems, which becomes especially appealing in combination with log transforms because it lets rules express clean products and ratios of variable powers.

The broader message is that a modest change to the atomic unit of a rule, from a single threshold to a sparse linear inequality, can ripple through an entire modeling framework, delivering simpler models at equal accuracy and reducing the field’s dependence on manual feature engineering. With a Python implementation publicly available on GitHub and a single interpretable complexity parameter governing the trade-off, LLTBoost lowers the barrier for practitioners who need models that both perform well and can be read, checked, and defended by the people who use them. In an era when regulatory and ethical pressure increasingly demands explainable automated decisions, techniques that shrink the accuracy gap between transparent and opaque models may prove among the most consequential tools in applied machine learning.

Subject of Research: Sparse oblique rule boosting for interpretable additive rule ensembles in machine learning

Article Title: Sparse oblique rule boosting for simpler additive rule ensembles

Article References: Behzadimanesh, S., Le Bodic, P., Webb, G. I., & Boley, M. (2026). Sparse oblique rule boosting for simpler additive rule ensembles. Data Mining and Knowledge Discovery, 40(6), Article 109. https://doi.org/10.1007/s10618-026-01241-8

Image Credits: AI Generated

DOI: 10.1007/s10618-026-01241-8

Keywords: machine learning, interpretable AI, gradient boosting, rule learning, additive rule ensembles, sparse linear transformations, oblique decision boundaries, data mining, statistical learning, model complexity, feature engineering, LLTBoost