Machine learning reveals investor demand as the main driver of Malaysian IPO underpricing

When a company floats its shares on a stock exchange and the price leaps on the first day of trading, the issuer has effectively left money on the table. This phenomenon, known as initial public offering underpricing, is one of the most stubborn anomalies in finance, and it is particularly extreme in markets where offer prices are fixed before anyone knows what investors are willing to pay. Malaysia is a striking case: because of limited liquidity and the prohibitive cost of book-building, Malaysian IPOs are priced through a fixed-price mechanism, and the average first-day pop exceeds 50 percent. A new study published in Discover Artificial Intelligence has now brought a battery of machine learning tools to bear on this puzzle, using 350 Malaysian IPOs from 2004 to 2021 to identify which algorithms best predict underpricing and, more importantly, which economic forces actually drive it.

The research team, led by Ali Albada of Sohar University with collaborators from Oman, Malaysia and beyond, faced a methodological challenge that has long hampered the field. The most interesting dynamics in IPO pricing are moderation effects, in which the influence of one factor depends on the level of another. Conventional linear regression assumes that each feature contributes independently and additively, which makes it blind to such conditional relationships. The authors’ solution was twofold. First, they engineered explicit interaction terms, such as the product of investor demand and the knowledge gap, so that even linear models could learn non-linear moderation. Second, they applied SHAP, or Shapley Additive exPlanations, a game-theoretic interpretability technique that quantifies each feature’s marginal contribution to predictions and can expose threshold effects that a single regression coefficient would miss.

Seven algorithms were benchmarked in a rigorously controlled pipeline: linear regression, support vector regression, k-nearest neighbors, random forest, XGBoost, LightGBM and a multilayer perceptron. The dataset was split 80/20 into training and hold-out test sets, features were standardized using statistics computed only on the training data to prevent leakage, and hyperparameters were tuned by grid search with five-fold cross-validation. The verdict was surprising to anyone who assumes that bigger models are always better. The humble support vector regression with a linear kernel won decisively, achieving a test R-squared of 0.512 and a cross-validated R-squared of 0.486, with a train-test gap of just 0.050, indicating minimal overfitting.

The glamorous ensembles fared far worse out of sample. XGBoost posted the highest training R-squared at 0.624 but lost 0.226 of that advantage on the test set, while random forest and LightGBM showed similar overfitting gaps of 0.166 and 0.221. With only 350 observations, the flexibility that makes tree ensembles powerful becomes a liability: each fold of cross-validation produces divergent trees, inflating variance. Formal Diebold-Mariano tests confirmed that the linear SVR’s advantage over random forest, XGBoost and the neural network was statistically significant at the five percent level, and out-of-time validation, training on 2004 to 2015 and testing on 2016 to 2021, showed the model retained 94 percent of its performance, ruling out period-specific artifacts. The message is clear: in small, noisy financial datasets, carefully regularized linear models with smart feature engineering beat complexity.

Having settled the predictive question, the team turned to interpretation, retaining the random forest specifically for SHAP-based interaction diagnostics. The global feature importance rankings converged across three independent methods, SHAP values, an ablation study and SVR coefficients, on a short list of dominant drivers. The over-subscription ratio, a measure of investor demand recorded days before listing, contributed 40.7 percent of predictive power and, on its own, accounted for 66.3 percent of the model’s explanatory capacity. The knowledge gap, an ex-ante uncertainty proxy capturing information asymmetry between issuers and investors, contributed another 15.6 percent. Together with market condition and offer price, the top five features explained 88 percent of the model’s predictions, a remarkably parsimonious result.

The most consequential finding concerns how demand and information asymmetry interact. Hierarchical regression with bootstrapped confidence intervals, run over 10,000 resamples, confirmed a significant positive interaction between the over-subscription ratio and the knowledge gap, with a coefficient of 0.0284 and a p-value of 0.002 that survived Bonferroni correction for three simultaneous hypothesis tests. But SHAP dependence plots revealed something a single interaction coefficient cannot: the moderation is non-linear. At moderate demand levels, between 10 and 30 times oversubscription, the knowledge gap’s effect on underpricing is strong. Above 30 times, it collapses by 73 percent, dropping from a coefficient of 0.067 to 0.018.

This threshold pattern maps directly onto Ivo Welch’s 1992 cascade, or bandwagon, theory. When oversubscription is extreme, herding takes over: investors stop evaluating fundamentals and simply follow the crowd, so information asymmetry stops mattering. Momentum-based decision-making overrides valuation concerns, and the price outcome is driven by the cascade itself. The authors are careful to frame this as a conditional association rather than proof of causation, acknowledging that the over-subscription ratio is simultaneously a revealed demand signal and a reflection of underpricing expectations. They treat this simultaneity as a theoretical feature of the cascading mechanism rather than a confound, noting that valid instrumental variables are scarce in the Malaysian setting.

The other two hypotheses fared less well. Offer price showed only marginal moderation of the knowledge gap effect, with a coefficient of 0.0198 and a p-value of 0.073 that did not survive multiple-testing correction, possibly because underwriters strategically set low prices when they anticipate high information asymmetry, creating reverse causality that muddies the signal. Listing board, comparing the Main Market with the riskier ACE Market, showed no moderation at all, with a negligible coefficient of 0.0089 and a p-value of 0.496. The likely explanation is that board membership is already encoded in correlated features such as offer price and offer size, leaving it no incremental predictive role once those covariates are controlled.

The study is not without limits, which the authors catalogue candidly. The best model still leaves roughly 49 percent of underpricing variance unexplained, hinting at omitted determinants such as founder characteristics, venture capital backing, media sentiment and ESG performance. The knowledge gap proxy inevitably blends information asymmetry with firm quality and ex-ante risk, constructs that are theoretically intertwined and difficult to decompose. And the findings are specific to a fixed-price regime; in book-building markets like the United States or United Kingdom, institutional behavior, analyst coverage and roadshows may dominate where retail demand rules in Malaysia.

Even so, the practical implications are pointed. For issuers and underwriters, managing demand through allocation policy matters more than almost any pricing signal. For investors, extreme oversubscription is both an opportunity and a warning that fundamentals have been sidelined by herding. For regulators, the concentration of predictive power in retail demand suggests that subscription allocation and disclosure rules directly shape pricing efficiency. Methodologically, the paper offers a template for financial machine learning: benchmark broadly, validate temporally, separate prediction from interpretation, and never trust a black box without bootstrapped statistical confirmation. Explainable AI, the authors argue, is most valuable not as a replacement for econometrics but as a pattern-finding partner that tells traditional inference where to look.

Subject of Research: Machine learning analysis of IPO underpricing drivers in Malaysia's fixed-price market

Article Title: Uncovering the drivers of IPO underpricing in Malaysia through machine learning benchmarking

Article References: Albada, A., Ramadan, R. A., Abusham, E. E., Ng, S. H., Ong, C. Z., Low, S.-W., & Qatiti, K. A. (2026). Uncovering the drivers of IPO underpricing in Malaysia through machine learning benchmarking. Discover Artificial Intelligence, 6(1), Article 1305. https://doi.org/10.1007/s44163-026-02390-x

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02390-x

Keywords: IPO underpricing, machine learning, SHAP, support vector regression, random forest, over-subscription ratio, information asymmetry, fixed-price mechanism, Bursa Malaysia, bandwagon effect, overfitting, explainable AI