The motivation behind the study reflects a broader problem in retention research. Much of the existing literature depends on proprietary industrial datasets, which reduces reproducibility, transparency, and comparability across studies. When the underlying data cannot be inspected, other researchers cannot verify how features were engineered or how models were reviewed, making it difficult to distinguish genuine methodological advances from results that simply cannot be checked. The researchers argue that publicly available benchmarks are the antidote. Their choice was the MovieLens 1M dataset, compiled by the GroupLens research group, a widely used resource for recommendation systems and user modeling research. By constructing retention models on an open benchmark with clearly documented structure, the team aimed to establish a standardized setting in which predictive techniques could be fairly compared and independently replicated, addressing a gap that has long limited progress in churn and engagement modeling.

The dataset itself is substantial. MovieLens 1M contains 1,000,209 explicit ratings from 6040 users on 3952 movies, collected between 2000 and 2003. Every user in the dataset had rated at least 20 movies, ensuring a minimum level of engagement and sufficient behavioral history for analysis. The ratings file records the user ID, movie ID, rating value on a one-to-five scale, and the Unix timestamp when each rating was posted, allowing interactions to be tracked over time. To define retention, the researchers applied a 30-day inactivity threshold: a user whose gap between their last two interactions exceeded 30 days was labeled as likely churned, while users remaining active within that window were labeled retained. The resulting classification problem was moderately imbalanced, with 3612 users, or 59.8 percent, categorized as retained and 2428 users, or 40.2 percent, as churned. The team noted that this imbalance, while not severe, could allow a classifier to achieve high accuracy simply by predicting the more common retained class, so the class distribution was carefully considered when selecting evaluation metrics and interpreting results.

Feature engineering formed the analytical heart of the framework. The researchers transformed raw interaction records into four families of variables: behavioral, temporal, content-based, and demographic. Behavioral features included each user’s mean rating and rating variance, capturing how consistent a user’s evaluations were over time. Temporal features proved especially important, including rating frequency, defined as the number of ratings divided by the user’s active time span, and the average inter-rating time, where longer intervals between interactions may signal waning interest. Activity recency, measured as the number of days since a user’s last interaction, and a binge index, defined as the maximum number of ratings posted in a single day, rounded out the temporal picture. Content-based features captured genre preferences through a genre affinity vector, with genre diversity quantified using entropy over the distribution of genres each user rated. Demographic variables, including age group, gender, and occupation, were one-hot encoded for the models. All numerical features were scaled using min-max normalization, and Pearson correlation analysis was conducted to examine redundancy among highly correlated features such as mean rating and rating variance.

With the feature table assembled, the researchers trained and compared an unusually broad selection of ten algorithms. Logistic Regression served as a linear baseline, estimating retention probability through a sigmoid function, while a Decision Tree provided a simple nonlinear classifier that divides the feature space using thresholds that reduce class impurity. A Support Vector Machine with a radial basis function kernel tested whether margin-based learning improves retention classification, and a Random Forest, which builds multiple decision trees on bootstrap samples and combines their predictions through majority voting, reduced the overfitting risk of a single tree. A Multilayer Perceptron acted as the neural network baseline, learning nonlinear relationships through hidden layers and activation functions. Four gradient boosting variants followed: classical Gradient Boosting, which builds trees sequentially to correct previous errors; XGBoost, which adds regularization terms and efficient tree pruning; LightGBM, which grows trees leaf-wise for computational efficiency; and CatBoost, which is optimized for handling categorical features such as gender, occupation, and genre attributes.

The tenth model, TabPFN 2.5, is fundamentally different from the boosting family. Rather than being a tree ensemble, it is a tabular foundation model that relies on prior knowledge acquired from a vast repository of synthetic tabular tasks and applies that knowledge to inference on a new dataset. The researchers included it to determine whether a recent foundation model could improve retention prediction on engineered tabular features. Training followed a rigorous protocol: the dataset was split 80/20 into training and testing subsets, with stratified 5-fold cross-validation applied to the training set to maintain the proportion of retained and churned users in each fold. Critically, the data split was performed in a time-aware manner rather than by randomizing past and future interactions. Features were built using only interactions observed within an observation window, and the retention label reflected behavior in the subsequent 30 days, a design intended to reduce information leakage. Hyperparameters were optimized through grid search with cross-validation, and all experiments were repeated with the same random seed for reproducibility.

The results confirmed the promise of the newest approach. On the held-out test set, TabPFN 2.5 achieved the highest scores across every metric reported: an accuracy of 0.814, a precision of 0.789, a recall of 0.741, an F1-score of 0.764, an AUC of 0.866, and a specificity of 0.862. Classical Gradient Boosting remained the strongest interpretable conventional ensemble model, with an accuracy of 80.2 percent, an F1-score of 0.752, and an AUC of 0.852, closely followed by XGBoost, LightGBM, and CatBoost. Statistical significance analysis based on cross-validation folds, with AUC as the main comparison, showed that Gradient Boosting significantly outperformed the simpler algorithms, including Logistic Regression, Decision Tree, Support Vector Machine, and Random Forest, but was not significantly different from the Multilayer Perceptron or the optimized boosting variants. TabPFN 2.5, however, was statistically significantly better than Gradient Boosting, consistent with the test-set results. ROC curves and precision-recall curves reinforced the same hierarchy, with TabPFN 2.5 and XGBoost showing the best overall precision-recall balance, and normalized confusion matrices showing TabPFN 2.5 achieving the strongest class-wise performance.

Feature importance analysis, conducted primarily through the interpretable Gradient Boosting model, revealed which variables actually drive retention predictions. Behavioral and time-related variables dominated: rating frequency, inter-rating intervals, and genre diversity emerged as the strongest predictors, with activity recency also highly influential. Users who were retained tended to rate more often, had shorter intervals between interactions, and showed more diverse genre engagement, particularly in drama and action. Descriptive statistics showed that retained users demonstrated more consistent and diverse rating behavior and shorter interactivity gaps than churned users. Demographic factors carried some predictive power, with younger users and those in student or technical occupations more likely to remain, and female users showing a slightly higher churn rate than male users, but the effects of gender and location were comparatively weak. The researchers interpreted these patterns across model families, noting that linear models capture additive effects for frequency and recency, while tree-based and boosting models handle threshold behaviors and nonlinear interactions. Sensitivity experiments with inactivity thresholds of 7, 14, and 60 days showed that absolute performance depends on the threshold chosen, though the relative positions of the models remained stable, with Gradient Boosting consistently among the strongest.

The authors are careful about what these findings mean. Because the MovieLens 1M dataset reflects historical user interactions from two decades ago, the results should be interpreted primarily as a reproducible methodological baseline for retention modeling rather than a direct representation of current media-platform behavior or contemporary consumption patterns. The study’s principal contribution, the researchers emphasize, is not a novel algorithm but a transparent framework built on open data, demonstrating that retention prediction can be rigorously tested on a public benchmark with understandable features and standardized experimental conditions. Future work could enrich the models with watch time or click data from streaming platforms, apply sequence models such as LSTMs or Transformers to time-varying user actions, and test generalization on data from contemporary services. Ablation studies could clarify the contribution of different feature groups. For media companies, the practical implication is clear: how users interact with a platform, in frequency, consistency, and variety, says far more about whether they will stay than who they are, meaning behavioral monitoring and timely engagement interventions may curb churn-related losses more effectively than demographic targeting.

Subject of Research: Predictive modeling of user retention from online behavior using the MovieLens 1M dataset and ten machine learning models

Article Title: Predictive modeling of user retention based on online behavior in media service platforms

Article References: Chen, J., & Chen, D. (2026). Predictive modeling of user retention based on online behavior in media service platforms. Discover Artificial Intelligence, 6(1), Article 1008. https://doi.org/10.1007/s44163-026-02043-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02043-z

Keywords: user retention, churn prediction, machine learning, MovieLens, TabPFN, gradient boosting, media platforms, behavioral data, feature engineering, reproducibility, tabular foundation models, Discover Artificial Intelligence