GPT-4o Ranks Freight Cancellations Well but Fails the Cost Test

A large language model can rank shipping-cancelation risks just as well as specialized statistical tools, yet still lose money the moment a company acts on its predictions. That paradox sits at the heart of a new study in Machine Learning with Applications by Wenyi Kuang, who builds a cost-sensitive architecture audit and deploys it against a real-world freight cancelation problem at a large U.S. online retailer. The research examines 354,565 truckload shipments and finds that prompted GPT-4o, despite achieving a competitive area under the ROC curve of 0.810, flags every single shipment for intervention because its lowest predicted probability never falls below the firm’s cost-implied decision threshold of 0.091. The result is an action policy identical to the crude strategy of always intervening, plus an extra $1.80 per thousand shipments in API costs.

The underlying business problem is one of asymmetric error costs. Every morning, freight planners must decide which scheduled shipments warrant pre-execution intervention, such as reconfirming the carrier, staging backup capacity, splitting the load, or warning downstream customers. A false alarm, intervening on a shipment that would never have been canceled, costs roughly $35 in coordination effort. A missed cancelation triggers last-minute re-tendering, delays, and service disruption at roughly $350 per shipment. That 10-to-1 asymmetry implies a mathematically optimal decision threshold of p* = 0.091: any shipment whose predicted cancelation probability exceeds 0.091 should be flagged, and anything below it should be released. Classical decision theory going back to Elkan’s foundational work on cost-sensitive learning prescribes exactly this kind of threshold, but the study shows that whether a model can actually use such a threshold depends on a property almost nobody measures: the range of probabilities the model is capable of producing.

Kuang formalizes this insight through four diagnostics that together decompose an architecture’s deployment value into four separable components: the information it consumes, the support and calibration of its predicted probabilities, the deployment policy, and the marginal inference cost. The first diagnostic asks whether competing architectures rank instances equally well when given identical features. The second, and the sharpest, detects a prediction-threshold mismatch: when a model’s score distribution lies entirely on one side of the decision threshold, the model degenerates into a constant-action rule no matter how well it ranks. The third evaluates routing architectures, testing whether selectively escalating hard cases to an expensive model beats a cheap deterministic fallback. The fourth probes behavior under a fixed intervention budget, where ties in the score distribution can silently destroy precision.

The empirical comparison was designed with unusual rigor. All architectures, including the language model, were restricted to the same 23-field audit record: ten categorical fields such as origin, destination, and carrier code, five numeric attributes such as mileage and planned transit hours, and eight engineered empirical-Bayes shrinkage priors estimated from historical cancelation rates by carrier and lane. The tabular learners, logistic regression, XGBoost, LightGBM, and an equal-weight ensemble, were trained on 266,915 shipments, calibrated on a separate 3,914-shipment fold, and scored once on a held-out set of 15,000 shipments. GPT-4o was never trained on any shipment; it was a frozen pretrained model queried once per shipment with the same features serialized into a prompt demanding a JSON cancelation probability. On ranking, the language model was statistically indistinguishable from the boosted trees, with paired bootstrap confidence intervals placing its AUC gaps to XGBoost (+0.002) and LightGBM (-0.002) inside a pre-specified equivalence band of plus or minus 0.02.

Ranking parity, however, masked what the author calls action collapse. GPT-4o’s predicted probabilities took only 96 distinct values across 15,000 shipments, and the lowest was 0.150, well above the operating threshold of 0.091. Consequently the threshold rule flagged 100 percent of shipments, matching the always-intervene baseline pointwise. Once the $1.80-per-thousand API cost was added, the language model architecture underperformed the free always-flag rule by about $2 per thousand shipments. The same-information learners avoided this trap because their continuous score distributions straddled the threshold. A raw-field logistic regression flagged only 91 percent of shipments and saved $394 per thousand against the baseline, the largest margin in the study, followed by the full-feature logistic model, the ensemble, the priors-only logistic model, and LightGBM.

Post-hoc calibration, the standard remedy for miscalibrated probabilities, did not rescue the language model. Isotonic regression improved the Brier score but left every probability above the threshold. Platt scaling and Beta calibration did push some scores below 0.091, yet they made operational cost worse, because the shipments they released included an outsized share of genuine cancelations whose $350 miss penalties overwhelmed the false-alarm savings. The study’s cost-ratio sweep shows the collapse is regime-dependent rather than universal: whenever the false-negative-to-false-positive ratio exceeds roughly 5.67 to 1, placing the threshold below the model’s 0.150 support floor, the always-flag degeneration is mechanically guaranteed. At lower ratios, such as 4:1, continuous-score learners still dominated, and across a 400-cell grid of cost pairs the ensemble and the raw-field logistic model split all winning cells, with the language model winning none.

An information ablation traced where the model’s apparent skill originated. An extraction regression of GPT-4o’s textual probabilities onto the prompt fields showed that raw numerics explained only 6.4 percent of output variance under cross-validation, while hashed categorical identifiers alone explained 87.5 percent and the eight engineered priors alone explained 88.2 percent, saturating at 89.2 percent with everything supplied. The language model was essentially re-expressing the firm’s own historical entity-level cancelation statistics back to it, not extracting novel signal from a pretrained world model. Stripping the priors from the prompt collapsed its discrimination to an AUC of 0.508, essentially chance.

The audit also exposed a subtle failure in popular routing architectures. A natural deployment routes sparse-history shipments, carriers appearing fewer than K times in the training data, to the language model while handling the rest with a calibrated tree. But because every routed shipment scored at or above 0.15, escalation and an unconditional flag-everything fallback produced identical actions on the routed subset at every value of K tested. Under any positive per-call API charge, the free deterministic rule strictly dominates, a result the paper proves formally and verifies empirically across the full routing grid. A capacity stress test added a further wrinkle: under a strict Top-500 intervention budget, the language model’s discrete score lattice forced 476 of the selected shipments to share the cutoff score, so precision was determined by the tie bin’s base cancellation rate of 97.1 percent rather than any within-bin ranking, while continuous-score architectures retained more decision-relevant resolution.

The central finding survived out-of-time validation. On a strictly chronological fold of 67,938 shipments from April to July 2021, split into four consecutive quartiles, the language model continued to rank competitively, trailing the raw-field logistic model by 0.006 to 0.019 AUC with all bootstrap intervals strictly positive, yet it again flagged 100 percent of shipments in every quartile and delivered no cost improvement over always-flag. A bounded robustness check with Anthropic’s Claude Sonnet under identical prompts reproduced the collapse, and a 28-cell prompt factorial across both providers found that no elicitation condition produced probabilities reaching zero, with 22 of 28 conditions flagging at least 99.4 percent of the cohort. The author is careful to scope the claim: the result does not show that language models cannot rank tabular data, only that at this cost ratio, with this probability elicitation interface and the observed score support, the model never changes the action the firm would take anyway. The ranking would likely flip under a lower cost ratio, a calibrated single-token log-probability interface, or richer unstructured inputs unavailable to the tabular baselines.

The transferable contribution is methodological rather than a verdict on any model class. Because deployment value depends jointly on information, score support, policy, and inference cost, a fair architecture comparison must audit whether a model’s scores actually cross the decision threshold and change actions, not merely whether it ranks well. In this freight application, a plain logistic regression on raw fields delivered the cheapest action policy, while an expensive frontier model reproduced a free baseline at a loss, a cautionary tale for any operations team tempted to equate benchmark scores with business value.

Subject of Research: Cost-sensitive auditing of large language model architectures for tabular prediction in freight cancellation management.

Article Title: COST-SENSITIVE ARCHITECTURE AUDITING FOR LLM-BASED TABULAR PREDICTION: EVIDENCE FROM FREIGHT CANCELLATION MANAGEMENT

Article References: Kuang, W. (2026). Cost-sensitive architecture auditing for llm-based tabular prediction: Evidence from freight cancellation management. Machine Learning with Applications, 25, Article 100997. https://doi.org/10.1016/j.mlwa.2026.100997

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.100997

Keywords: large language models, GPT-4o, tabular prediction, cost-sensitive learning, freight cancellation, XGBoost, calibration, decision thresholds, logistic regression, architecture audit, machine learning deployment, supply chain