Synthetic Japanese claims benchmark puts healthcare fraud detection to a frozen, one-shot test

Healthcare fraud research has long been trapped in a paradox: the claim-level records and confirmed fraud labels needed to train and test anti-fraud machine learning are exactly the materials that privacy rules and investigation secrecy keep out of reach. A team led by Amir Hossein Alikhah Mishamandani, with Citadel colleagues Donna Canceran, Buqin Hu and Dipanjan Bhui, has now built a way around the impasse—at least for method development. Writing in the International Journal of Data Science and Analytics, they present a synthetic, relational, longitudinal benchmark of medical claims in which every fraud mechanism, every clinical trajectory and every label is generated by design, so that researchers can audit precisely what a detection method is exploiting. The benchmark is calibrated to Japanese public statistics, and its headline evaluation is a deliberately frozen, single-access test that the authors committed to before generating the data.

The word “calibrated” carries a deliberately narrow meaning here. Selected distributions—patient age and sex, prefecture, monthly utilization, provider type, hospital bed counts and diagnosis-procedure reimbursement capability—are constrained by documented aggregate references from Japan’s e-Stat portal and the Ministry of Health, Labour and Welfare. Coding and billing rules draw on ICD resources, medical-action and drug masters from the Social Insurance Medical Fee Payment Fund, and fee lookup tables. But the authors are explicit about what this does not imply: joint clinical histories, fraud labels, fraud prevalence and provider identities are simulated under versioned assumptions, not learned from those aggregate assets. The benchmark is a controlled experimental substrate, not a reconstruction of confidential Japanese claim histories, and it cannot estimate how much fraud actually occurs in Japan.

Structurally, the benchmark is far more than a flat classification table. Each of 50,000 synthetic patients is linked to 150,000 clean episodes, and claims are produced in dependency order: primary and secondary diagnoses are assigned before procedures, drugs and DPC reimbursement rows, with every choice conditioned on age, sex, chronic state, comorbidity, risk, care setting, provider specialty and prior treatment history. Longitudinal state permits compatible continuation, switching, discontinuation and escalation, and stable patient-level heterogeneity means that two patients sharing a diagnosis and provider need not receive identical care. Crucially, the generator also creates legitimate hard negatives—repeat testing, complex inpatient care, compatible high-cost treatment and chronic polypharmacy explicitly marked as non-fraud—so that a detection method cannot win by simply flagging anything expensive or unusual.

Providers are generated separately from patients through a process the authors call de-linking. Rather than emitting a renamed public facility row, the generator forms donor pools by provider type, prefecture and bed-count band, then independently resamples municipality, perturbs bed count, and derives capability, volume and structural-risk attributes before assigning synthetic identifiers and names. A hard postcondition forbids any exact match on the public-source signature of provider type, prefecture, city, bed count and specialty count. Release diagnostics compare all synthetic providers against the full public facility reference and found zero exact signatures across 7,640 providers—yet nearest-distance warnings remain, and the authors refuse to convert de-linking into a claim of anonymity.

The fraud layer is built for financial reconciliation. Nine mechanisms are configured—ordinary duplicate billing, near-duplicate claims, upcoding, excessive testing, drug overprescribing, DPC severity upcoding, phantom billing, unbundling and provider collusion—with subtypes distinguishing exact, same-family and adjacent-month duplicates, medication variants, bundle decomposition, artificial short episodes, provider peer outliers and shared-patient clusters. Every successful mutation carries a stable identifier and source-claim linkage, and monetary effects are independently recomputed from the clean and injected relational tables rather than accepted from the mutation request. The configured fraud parameter is a requested mutation-event allocation rate, not a guaranteed claim prevalence; insertion and multi-claim mechanisms make the two quantities differ by design, and neither estimates real-world fraud.

Perhaps the most striking part of the paper is what the authors retracted. An earlier injection path had written two fraud-descriptor literals—phantom_short_episode and adjacent_month_duplicate_episode—into a model-visible field called episode_type. Because these values appeared only on injected claims, any model could read the answer directly from the data: a textbook case of target-proxy leakage. The team withdrew every result fitted under that contract, including their former principal result, and rebuilt all reported analyses after correction. A fail-closed provenance audit now compares clean and injected vocabularies for every categorical model input and blocks injection-only values by default. In the corrected confirmation, 760 values across 11 audited categorical fields were assessed, no model-visible value was blocked, and exact scans of the 75-feature matrix and its 218 fitted names found no textual leakage—though the authors caution that this cannot prove a designed mechanism lacks highly separable observable proxies.

The evaluation protocol is engineered to prevent subtle forms of self-deception. A new scenario, seed 202611, was generated after a precommitted protocol bound the temporal split, selection metric, model candidates, calibration method, threshold rule and reporting endpoints. Model family and hyperparameters were chosen solely by nested temporal cross-validation on development data, using the absolute amount of synthetic yen recovered at a 1% review budget as the objective. LightGBM won that development contest, recovering JPY 24,071,578 on average across outer folds—notably, histogram gradient boosting had higher development average precision but a lower selection value, showing that ranking quality and operational recovery are not interchangeable. Platt calibration and a cost-sensitive operating threshold were then fitted on a separate pre-final interval, and the frozen pipeline was serialized before the evaluation code was permitted its single, ledgered access to the final partition.

That one evaluation produced the paper’s principal result. On 53,609 final-period claims containing 1,573 designed positives, the frozen LightGBM pipeline achieved a non-interpolated average precision of 0.8403 (95% bootstrap confidence interval 0.8266 to 0.8545), a ROC area under the curve of 0.9764, a Brier score of 0.01509 and an expected calibration error of 0.03375. At the precommitted 1% review budget, the top 537 selected claims were all true positives with no false positives, yielding recall of 0.3414 and JPY 128,752,597 in injected amount recovered. A separate cost-sensitive threshold, frozen at 0.69296, traded precision for coverage: 1,188 true positives against 262 false positives, recall of 0.7552 and JPY 140,217,534 recovered. The authors stress that these yen figures are sums of synthetic counterfactual deltas under benchmark costs, not estimates of collectible real-world savings.

Corrected historical diagnostics—kept separate because their final partitions had prior campaign access—reveal how unevenly difficult the designed mechanisms are. Average precision ranged from 0.070 for duplicate billing, 0.079 for drug overprescribing and 0.093 for upcoding to 0.874 for DPC severity upcoding, 0.965 for provider collusion, 0.970 for phantom billing and 0.973 for unbundling. A development-only covariate-shift challenge cut average precision from 0.791 to 0.729, and a feature-family ablation showed that amount-only, graph-only or Japan-reference-calibrated-only views could not reproduce the full-feature ranking, with calibrated-reference-only features collapsing to average precision 0.0426. A clinically stratified provider-permutation comparator, which weakens longitudinal patient-provider continuity and graph structure while preserving clinical rows, actually scored higher than the structured same-seed row—a diagnostic result the authors interpret carefully as sensitivity to relationship construction rather than evidence that simpler generation is generally superior.

The limitations are stated with unusual candor. The confirmation is one internal synthetic scenario; the bootstrap interval remains conditional on one frozen generated dataset and captures neither entity dependence nor generator uncertainty. Only selected marginals are calibrated to public sources, some facility and fee snapshots post-date the simulated observation window of April 2024 through March 2025, and DPC point arithmetic applies a single selected period across the full stay rather than accumulating piecewise periods—an explicit modeling debt, alongside the numeric encoding of the nominal MDC category. The temporal estimand includes recurring patients and providers, so results say nothing about wholly unseen providers, and the software ledger documents process separation without claiming cryptographic or human blinding. What the benchmark offers instead is transparency: versioned assumptions, traceable fraud counterfactuals, auditable model selection and explicit failure modes. Strong synthetic performance, the authors conclude, is separability of declared mechanisms under declared budgets—and nothing more until authorized labeled data can support real external validation.

Subject of Research: A synthetic, Japan-calibrated relational medical claims benchmark with traceable fraud mechanisms for reproducible healthcare anti-fraud machine learning evaluation

Article Title: A Japan-calibrated synthetic medical claims benchmark for anti-fraud machine learning

Article References: Mishamandani, A. H. A., Canceran, C. D., Hu, B., & Bhui, D. (2026). A Japan-calibrated synthetic medical claims benchmark for anti-fraud machine learning. International Journal of Data Science and Analytics, 22(1), 361. https://doi.org/10.1007/s41060-026-01336-x

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01336-x

Keywords: synthetic health data, healthcare fraud, medical claims, machine learning benchmark, Japan, LightGBM, data leakage, model calibration, temporal cross-validation, provider de-linking, fraud detection, reproducibility