FDA Shortage Data Offers Uneven AI Warning and Policy Lessons

A public U.S. Food and Drug Administration database can help identify generic drugs that may face another shortage within 90 days, but a new study finds that the signal is most useful as a triage tool rather than an automatic forecasting system. In a temporal test of 1,326 monthly observations, gradient boosting ranked recurrence risk with a receiver-operating-characteristic area under the curve of 0.831, while logistic regression placed six of 15 future shortage events within the highest-risk 10% of observations. Those results suggest that artificial intelligence can concentrate a limited review team’s attention, yet most flagged cases remained false positives and the test included only 15 positive recurrence windows. The accompanying inventory experiment was conducted entirely in a synthetic environment and did not show that reinforcement learning was better overall than simpler rules.

The research analyzed 1,688 records from an FDA drug-shortage snapshot containing 252 unique generic names and 137 companies. It separated two questions that are easily conflated: whether a record currently has a shortage status, and whether a drug that has previously appeared in the shortage data will experience another event within the next 90 days. Current-status classification produced extremely strong results, with F1 scores between 0.983 and 0.991 and precision-recall area under the curve above 0.998, even after direct status-related fields were removed. The authors treated this as a diagnostic showing how structured the snapshot is, not as evidence that future shortages can be predicted accurately.

For the prospective task, records marked Current or Resolved were collapsed into monthly events for each generic drug. The models used only information available before the target month, including prior event counts, events during the previous 365 and 730 days, time since the last event, company and presentation counts, therapeutic category, and historical shortage-reason indicators. Training covered months through 2022, validation covered 2023–2024, and testing covered 2025–2026. Logistic regression, random forest, and gradient boosting were compared with a historical recurrence rule, while strict holdouts tested whether performance extended beyond particular generic names or companies.

Because only 1.13% of temporal test windows contained a recurrence event, conventional accuracy would be misleading. The study therefore emphasized the area under the receiver-operating-characteristic curve, area under the precision-recall curve, recall, capture within the highest-risk 10%, and Brier score. Gradient boosting achieved the best reported discrimination, with a precision-recall area under the curve of 0.0465 and recall of 0.667. That precision-recall value was about four times the event-prevalence baseline, but its absolute magnitude remained low. Logistic regression performed best for the operational triage measure, capturing 40% of positive windows in the top 10% of risk scores.

The distinction between ranking and prediction is critical to the findings. A team able to review one in 10 item-months would examine 133 of the 1,326 test observations, of which six corresponded to future recurrence events. That could be useful when the consequence of missing a shortage outweighs the cost of investigating an alert, but the precision of such a shortlist would remain modest. Thresholds would also need adjustment when the prevalence of shortage events changes, particularly during pandemics, recalls, or other demand shocks. The temporal test itself was small: the recurrence panel covered 78 generic names and included only 15 positive test windows, making threshold estimates and some performance measures uncertain.

The researchers also compared simple statistical methods for forecasting aggregate monthly shortage-event counts over a final 12-month interval. These methods did not perform the same task as item-level risk ranking, but they provided a useful check on whether more complex techniques were necessary. The six-month rolling mean produced the lowest root mean square error, 2.814, while the last-observation naive method produced the lowest symmetric mean absolute percentage error, 0.500. Exponential smoothing and an ARIMA(1,0,1) model performed worse on this small and irregular test, reinforcing the authors’ point that added model complexity should be supported by task-specific evidence rather than assumed to be beneficial.

Separately, the team tested tabular Q-learning against economic order quantity, min-max, safety-stock, risk-threshold, and human-rule inventory policies in a disclosed synthetic Markov decision process. The simulator used normalized quantities and a bounded risk proxy derived from FDA status; it did not use the prospective 90-day model probabilities. Its state combined binned inventory, risk, supplier reliability, and lead time, while available actions ranged from taking no action to increasing orders or safety stock, expediting purchases, conducting emergency procurement, and requesting human review. Supplier switching and product substitution were excluded because the public records contain no evidence on supplier capacity, contract prices, or clinical substitutability.

No inventory policy dominated every objective. Q-learning produced a stockout rate of 0.1603, compared with 0.1631 for the min-max policy, and raised fill rate from 0.8885 to 0.8982. Those service gains came with approximately 144 more units of total cost and an average of 7.61 emergency orders, whereas min-max generated no emergency orders in the reported comparison. The threshold policy achieved the lowest stockout rate, 0.0478, and the highest service level, 0.9522, but required the greatest number of emergency orders, human reviews, and safety interventions. The experiment therefore exposed trade-offs rather than a universally superior algorithm.

Robustness tests made the instability of the learned policy clearer. Q-learning’s stockout rate increased from 0.153 in a base sensitivity run to 0.283 when demand was raised by 50%, and its training rewards remained highly variable without stable monotonic convergence. Paired comparisons with min-max showed a small-to-moderate median stockout reduction of 0.05, with a sign-biserial effect of -0.245, and a median fill-rate increase of 0.045 with an effect of 0.412. These changes did not erase the higher costs and emergency ordering. The authors consequently characterize Q-learning as an exploratory comparator, not a converged optimal policy, and note that real deployment should favor the simplest policy capable of meeting safety and service requirements.

The human-in-the-loop elements were also tested as governance proxies rather than as evidence about real users. Programmed rules requiring human approval reduced the simulated rate of high-risk autonomy violations to zero and raised audit completeness to 0.966, while adding explanations improved audit records without removing every violation. These outcomes show that the coded controls worked mechanically; they do not establish that explanations improve trust, reduce cognitive burden, or create accountability among pharmacists and procurement staff. Descriptive diagnostics also showed large differences in current-status rates among therapeutic categories with at least 50 records, ranging from 0.141 in endocrinology and metabolism to 0.985 in anesthesia and pediatrics. Because the dataset has no patient demographics and the prospective test contains only 15 events, it cannot establish patient-level fairness.

The study’s clearest conclusion is that public shortage histories can support a carefully bounded warning and review system, but not autonomous purchasing or a definitive shortage declaration. A real implementation would need local demand, inventory, procurement, contract, supplier-capacity, lead-time, clinical-criticality, and formulary data, none of which are available in the FDA snapshot. It would also require integration with enterprise resource planning and pharmacy systems, identity and cybersecurity controls, monitoring for distribution shift, category-specific calibration, and prospective testing in silent mode. Until those steps are completed with hospital data and qualified users, the empirical results support risk triage, while the inventory and governance findings remain scenario-derived demonstrations of trade-offs rather than proof of clinical or operational benefit.

Subject of Research: Evaluation of AI-based drug-shortage recurrence ranking and synthetic inventory-policy simulation using public FDA records

Article Title: Human-in-the-loop artificial intelligence for drug-shortage recurrence-risk ranking and synthetic inventory-policy simulation using U.S. Food and Drug Administration Data

Article References: Khan, M. R. I., Al Mamun, A., & Tahsin, M. S. (2026). Human-in-the-loop artificial intelligence for drug-shortage recurrence-risk ranking and synthetic inventory-policy simulation using U.S. Food and Drug Administration Data. Discover Artificial Intelligence, 6(1), Article 1080. https://doi.org/10.1007/s44163-026-02144-9

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02144-9

Keywords: drug shortages, FDA data, artificial intelligence, machine learning, risk ranking, Q-learning, inventory policy, healthcare supply chains, human oversight, Human-in-the-loop, artificial, intelligence