Machine learning models may not encode cross-cultural lifestyle-health differences

When machine learning models trained on lifestyle data from Indian and Japanese university students appeared to reveal strikingly different health risk architectures, the finding seemed to confirm an increasingly popular narrative in digital health research: that machine learning can detect culturally specific patterns in how lifestyle factors drive disease. But a rigorous, leakage-free reanalysis published in Frontiers in Digital Health tells a very different story. After addressing three fundamental methodological problems in the preliminary analysis, the research team found that the entire India-Japan feature-importance gap was statistically indistinguishable from what an eightfold difference in sample size produces on its own.

The study began as a comparison of lifestyle and noncommunicable disease feature importance between Indian (n = 528) and Japanese (n = 70) university students, using cross-sectional data collected from June to August 2024. India and Japan present contrasting epidemiological and dietary profiles that made the comparison appealing. India faces a rapid nutrition transition, with sedentary behaviour independently associated with obesity risk among urban Indian adults, while Japan has undergone decades of continuous dietary westernization alongside a strong cultural link between lifestyle behaviour change and subjective well-being. Body mass index also carries different metabolic implications in South Asian populations than in Western reference populations, making BMI-based thresholds an imperfect, population-dependent proxy for adiposity rather than a universal standard.

But the preliminary analysis contained three specific, checkable pitfalls that the authors identified in their own work. First, the outcome of obesity, defined as BMI ≥ 25 kg/m², was tautologically predicted by including BMI itself as a model input. Second, model evaluation was performed without a held-out test set, meaning the models were evaluated on the same data they were trained on. Third, feature-importance comparisons ignored sampling variability from an eightfold difference in cohort size. The team addressed each problem using penalized logistic regression, Random Forest, and gradient-boosted trees with proper train/test splitting, held-out permutation importance, and bootstrap confidence intervals.

The most dramatic result came from quantifying the BMI-obesity circularity. With BMI included as a predictor, all three India-obesity models achieved perfect discrimination on the held-out test set, with an AUC of 1.000 for logistic regression, Random Forest, and XGBoost alike. BMI’s Gini importance was 0.91 with a bootstrap 95% confidence interval of [0.84, 0.92], and its held-out permutation importance was 0.50, meaning permuting BMI alone destroys half of the model’s discriminative AUC. This is definitional, not empirical: obesity is BMI ≥ 25 by construction. When BMI is removed from the predictor set, test AUC collapses to 0.568 for logistic regression, 0.526 for Random Forest, and 0.503 for XGBoost, indistinguishable from chance. Lifestyle features carry essentially no independent discriminative signal for obesity once the tautological predictor is removed.

For the remaining viable outcome, elevated self-rated stress, held-out predictive performance was modest at best. In India, test AUC was 0.67 for logistic regression, 0.67 for Random Forest, and 0.71 for XGBoost, with a bootstrap 95% confidence interval of [0.49, 0.72]. In Japan, the models did not clearly exceed chance, with AUCs of 0.57, 0.57, and 0.49, and a very wide bootstrap confidence interval of [0.29, 0.82] reflecting Japan’s small test set of only 14 observations. None of the AUC values in the 0.92 to 0.98 range obtained under the preliminary, non-held-out evaluation approach were reproduced once a genuine held-out test set was used.

The reanalysis also revealed a critical discrepancy in how feature importance was measured. Gini importance, computed on the training fit, reflects how much a variable reduced impurity during tree construction, while permutation importance on held-out data reflects how much actual predictive information the model loses without that variable. These two measures frequently disagreed. Most notably, sleep duration’s Gini importance in Japan was 0.22, superficially close to the 0.47 value initially reported as headline evidence for a Japanese psychosocial phenotype, but its held-out permutation importance was −0.004, indicating no measurable out-of-sample discriminative contribution at all. By contrast, BMI with a permutation importance of 0.067 and exercise frequency at 0.040 contributed more to real, held-out predictive performance than sleep duration did. The claim that sleep duration is the primary stress predictor in Japan is not supported once importance is assessed out-of-sample.

The study’s central contribution is a sample-size sensitivity analysis that directly tests whether the India-Japan feature-importance difference exceeds what unequal cohort size alone predicts. The team repeatedly subsampled the Indian cohort, without replacement, down to Japan’s analytic sample size of 70, refitting the Random Forest pipeline on each of 300 draws and recording the resulting distribution of feature importances. For every feature examined, including age, sex, BMI, sleep duration, exercise frequency, water intake, and all food-preference indicators except one near-zero-importance category, Japan’s actual observed Gini importance value fell inside the 95% range produced by this subsampling procedure alone. In other words, drawing a random subsample of 70 Indian participants and fitting the identical pipeline would routinely observe importance values as different from India’s full-sample estimate as Japan’s actual values are.

The reanalysis also uncovered a measurement-instrument mislabeling that the authors disclose transparently. Perceived stress was intended to be assessed using the Perceived Stress Scale (PSS-10), a validated 10-item instrument. But on auditing the raw survey exports, no PSS-10 items exist in either dataset. The actual instrument administered at both sites was a single self-rated item asking participants to rate their stress or anxiety level on a scale of 0 to 10, which was binarized at 5 or above to define elevated stress. A single-item stress rating has not been validated for cross-cultural measurement equivalence, and the authors treat all stress results as exploratory accordingly. Similarly, once the study’s own chronic-illness exclusion criterion was correctly applied, positive diabetes cases fell to zero in both cohorts, and Japan’s survey instrument never asked a hypertension-related question at all.

The authors emphasize that both cohorts are convenience samples recruited from a single institution in each country with substantially different response patterns. India retained 54.0% of initial respondents as age-eligible and not excluded, while Japan retained only 20.2%. Any apparent difference between them may reflect institutional, disciplinary, or self-selection factors as readily as national culture. The authors also note that they did not conduct formal cross-cultural measurement-invariance testing for any instrument used in the study, meaning that interpreting comparable constructs across sites remains a working assumption rather than a demonstrated fact. Self-reported height and weight typically underestimate true BMI, introducing further measurement uncertainty.

Despite the negative findings, the authors regard the analysis as a legitimate and useful contribution in its own right. The pattern illustrated here, including small paired cohorts, no held-out evaluation, circular outcome definitions, Gini importance treated as ground truth, and an oversampling step that was described but never actually implemented, is not unique to this dataset. The checks the authors describe, including quantifying circularity by ablating the offending predictor, computing held-out permutation importance alongside Gini importance, and running a sample-size-matched subsampling sensitivity analysis, are general-purpose and inexpensive to apply before any cross-cultural machine learning feature-importance claim is published. The authors suggest that future studies routinely report an explicit list of which outcome-defining variables are excluded from the predictor set, held-out permutation importance alongside any impurity-based importance measure, and a sample-size-matched subsampling sensitivity check before interpreting any between-population difference as substantive. A genuine test of cross-cultural machine learning transferability would train a model on one cohort and evaluate its discrimination and calibration on the other, directly measuring transfer performance rather than comparing independently fitted feature importances. With complete, runnable code accompanying the paper, every number in the study can be independently verified, and the authors hope this transparency will help digital health research avoid mistaking statistical artifacts for cultural insight.

Subject of Research: Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan

Article Title: Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan

Article References: Takaoka, E., Ruban, S., Mohammed Siraj, B., Vinay, V., Shanthini, H., & Dias, A. (2026). Do machine learning models really encode cross-cultural lifestyle-health differences? A leakage-free reanalysis of two university cohorts in India and Japan. Frontiers in Digital Health, 8, Article 1791066. https://doi.org/10.3389/fdgth.2026.1791066

Image Credits: AI Generated

DOI: 10.3389/fdgth.2026.1791066

Keywords: machine, learning, models, really, encode, cross-cultural, lifestyle-health, differences, leakage-free, reanalysis, university, cohorts