Prompt-engineered embeddings fail to beat strong tabular models in demand forecasting benchmark

Demand forecasting sits at the heart of inventory management, because every unit of forecast error ripples directly into replenishment timing, safety stock, service levels and operating cost. A new benchmark study now offers a sober answer to one of the more fashionable questions in applied forecasting: does converting structured business data into natural-language prompts, and then into machine-learned embeddings, actually improve predictions? In a multi-store inventory setting, the answer was no for the strongest models tested.

The study, published in Discover Analytics, built a hybrid forecasting framework that transformed each row of a retail demand dataset into a standardized textual description of store, product, price, promotion, inventory, weather, epidemic and seasonal conditions. Those prompts were encoded into dense sentence embeddings, compressed with principal component analysis, and appended to conventional engineered time-series features. The resulting hybrid models were then pitted against tabular-only versions of the same algorithms under a strict chronological holdout protocol.

The data came from a publicly available Kaggle dataset uploaded by Ramin Huseyn, containing 76,000 observations across 760 daily timestamps, 5 stores and 20 products, with 16 variables. After lag-based feature engineering removed early periods without sufficient history, the final modeling dataset contained 73,200 observations, split into 58,500 training and 14,700 test rows. The authors are careful to note that the dataset’s original organizational provenance cannot be independently verified, so the findings should be read as dataset-specific and exploratory.

Three regression families were evaluated: LightGBM, XGBoost and Random Forest, each in a tabular form and a hybrid form augmented with 32 principal components derived from the sentence embeddings produced by the all-MiniLM-L6-v2 model. In the final holdout run, those 32 components retained 96.7159 percent of the training-embedding variance. Hyperparameters were tuned separately for each variant on a validation slice carved from the training period, using the same modest grid-search budget for all models.

The verdict was unambiguous. Tabular_XGBoost, the plain structured-data version of XGBoost, was the strongest tuned holdout model, achieving a mean absolute error of 11.4174, a root mean squared error of 16.3032, a mean absolute percentage error of 17.8859 percent, an R-squared of 0.8638 and a weighted absolute percentage error of 11.4220 percent. Hybrid contextual embeddings improved Random Forest, reducing its RMSE by 2.58 percent, but they worsened LightGBM by 1.19 percent and XGBoost by 9.10 percent in RMSE terms.

These within-family differences were not statistical noise. Paired bootstrap confidence intervals and Diebold-Mariano tests computed on final-holdout per-observation errors showed that the differences were statistically distinguishable. For LightGBM and XGBoost, the 95 percent paired bootstrap intervals for the RMSE difference were positive, indicating higher hybrid error, while for Random Forest the interval was negative, indicating lower hybrid error. The Diebold-Mariano tests on squared errors were significant for all three comparisons.

The benchmark was reinforced by two additional checks, both deliberately qualified. A model-seed stability check varied only the random seed across 7, 42 and 2024 while holding validation-selected hyperparameters and the embedding representation fixed, so it captures model-fitting stochasticity rather than full tuning uncertainty. Four expanding-window rolling-origin checks refit the models per window using the same validation-selected configurations; the authors describe these as descriptive temporal sensitivity checks rather than fully independent nested backtests. Under both checks, Tabular_XGBoost retained the lowest average RMSE.

Why did the prompt-engineered layer fail to help the boosting models? The most likely explanation, according to the authors, is that the prompts were generated from variables already present in the tabular feature matrix. The embedding therefore added no genuinely independent information source, only a redundant, higher-dimensional restatement of signals that tree-based learners can exploit directly. Feature importance analysis of the winning model supports this reading: the 7-period rolling mean of demand was the leading predictor, followed by the promotion-price interaction, promotion status, epidemic status and weather condition. Recent demand history remained the central source of predictive information.

The study also connects forecasts to operations through an illustrative inventory simulation with a fixed seven-day replenishment lead time. For each store-product series, beginning inventory was initialized from the first test observation, demand was fulfilled daily, and reorder targets were computed as seven-day forecast demand plus a safety-stock term of 1.65 times recent demand volatility times the square root of seven. The XGBoost variants produced the highest fill rates and lowest stockout-period rates, but the authors stress that the service-level differences are small and the simulation is a transparent decision-analytics illustration, not an optimized inventory-control policy or a basis for cost-ranking claims.

The broader lesson is a contingent view of prompt engineering in time-series analytics. Prompt-derived embeddings may provide selective value, as the Random Forest result shows, but they do not necessarily surpass strong structured-data benchmarks when operational predictors already contain the dominant forecasting signal. The authors suggest prompt engineering may prove more valuable when it introduces external text, manager notes or event descriptions not already captured in structured form. They also flag seven limitations, including the single dataset, one prompt template, one embedding model, a modest hyperparameter search and the simplified inventory layer, and call for future work with external unstructured text, alternative templates and richer inventory-control simulations.

Subject of Research: Benchmarking prompt-engineered contextual embeddings against tabular machine-learning models for multi-store demand forecasting

Article Title: Prompt-engineered contextual embeddings for panel time-series demand forecasting: a benchmark in multi-store inventory analytics

Article References: Khan, M. R. I., Mamun, A. A., Tahsin, M. S., & Shamsuddoha, M. (2026). Prompt-engineered contextual embeddings for panel time-series demand forecasting: a benchmark in multi-store inventory analytics. Discover Analytics, 4(1), Article 42. https://doi.org/10.1007/s44257-026-00094-1

Image Credits: AI Generated

DOI: 10.1007/s44257-026-00094-1

Keywords: demand forecasting, prompt engineering, contextual embeddings, XGBoost, LightGBM, Random Forest, time-series analytics, inventory management, sentence transformers, retail analytics, machine learning benchmark, supply chain