Small Language Models Turn Web Search Into Features for Tabular Data Classification

Tabular data remains the stubborn holdout of modern machine learning. While deep learning has swept through images, speech and text, spreadsheets of numbers and categories—the raw material of medicine, finance and industry—still resist neural approaches, and practitioners continue to rely on classical algorithms fed by carefully engineered features. A new open-access study in Scientific Reports by Emrah Inan and Damla Oguz of İzmir Institute of Technology in Türkiye proposes an unexpected bridge between these worlds: small language models that search the web, read what they find, and convert human knowledge into new columns for a classifier to learn from.

The feature space is the framework that turns each data sample into a vector a model can analyze, and the authors argue that the success of any machine learning pipeline depends heavily on how well important features are selected and generated. Feature generation, as a central technique, converts raw data into a refined feature space that facilitates model training and enhances its structure. Yet despite steady progress in automated feature selection and generation, two major challenges persist: explainability and cross-domain generalization. Methods that work brilliantly on one dataset often fail to transfer to another, and the features they produce can be opaque to the experts who must act on them.

One popular alternative has been to let deep learning handle feature engineering automatically rather than relying on hand-crafted techniques. Deep learning models have proven highly effective across a variety of fields, but they are often considered less effective when applied to tabular datasets, where sample sizes are modest, columns are heterogeneous, and the signal is spread across interactions that neural architectures capture inefficiently. The İzmir team therefore took a different route: instead of asking a giant network to internalize the data, they asked small language models to reach outside the dataset entirely and pull in knowledge from the web.

The methodology centers on retrieval-augmented generation, the technique that lets a language model consult external sources before answering. For each tabular dataset, the system formulates search queries derived from the column names, values and the classification task itself. A small language model—compact enough to run without massive infrastructure—then processes the retrieved web content and distills it into concrete, machine-readable features. These web-search-based features are appended to the original tabular columns, enriching each row with external context that no amount of resampling the raw data could have produced.

The choice of a small language model rather than a frontier-scale one is deliberate. Small models are cheaper to run, easier to deploy, and—crucially for the explainability problem—their prompts and retrieval steps can be inspected. The study also emphasizes prompt design as a first-class component of the pipeline: how the model is asked to read a web page and translate it into a feature determines whether the extracted signal is meaningful or noise. Because the resulting features are grounded in retrieved documents rather than in the latent whims of a neural network, a domain expert can, in principle, trace why a model believes a certain external fact is relevant.

Experimental results show the approach reaching highest F1 scores of 0.383 and 0.848 on two well-known tabular datasets for the machine learning multi-class classification task. The F1 score, the harmonic mean of precision and recall, is a demanding metric for multi-class problems, where a model must balance accuracy across every category rather than excelling on the majority class. The gap between the two scores reflects the reality of tabular classification: some datasets carry rich external context that web knowledge can illuminate, while others resist enrichment because their structure is idiosyncratic or their semantics are not well represented online.

The broader significance lies in what this says about the evolving role of language models in data science. Retrieval-augmented generation has mostly been framed as a question-answering technology—a way to make chatbots cite their sources. Inan and Oguz repurpose it as a feature factory. The web becomes a vast, continuously updated auxiliary dataset, and the small language model becomes the extraction engine that converts unstructured human knowledge into the structured columns that classical classifiers consume. This reframing matters because it suggests that the value of language models for tabular problems may lie not in replacing gradient-boosted trees, but in feeding them better inputs.

It also speaks to the open-source momentum in the field. The study’s keywords—feature selection, retrieval-augmented generation, small language models, open-source and prompt design—map onto a research agenda that many laboratories can pursue without access to proprietary giants. A team with a modest GPU and a careful prompt library can replicate the pipeline, adapt the search strategies to their domain, and audit every feature the system proposes. That transparency directly addresses the explainability gap the authors identify as a central challenge of automated feature engineering.

Cross-domain generalization remains the harder test. A feature generated from web content is only as good as the match between the dataset’s semantics and the public knowledge the search engine surfaces. Medical datasets may benefit from clinical literature, while proprietary industrial sensor logs may find little useful context online. The reported F1 scores, while promising on the stronger dataset, are an honest reminder that enriching tabular data with web knowledge is not a universal accelerant. The authors’ contribution is a methodology and a demonstration, not a guarantee—and they present it as an alternative to deep-learning-based automatic feature engineering rather than a replacement for domain expertise.

Still, the vision is arresting. Imagine a near future in which every spreadsheet arrives pre-enriched: a language model has already searched the web for what each column means, retrieved the relevant facts, and generated features that encode them. Classification models would train on data that carries not just measurements but context, and analysts would see, in plain retrieved documents, why each new column exists. Inan and Oguz have taken an early, concrete step toward that future, showing that small language models can select and generate web-search-based features for tabular classification—and that the humble spreadsheet may finally have found its companion in the age of language models.

Subject of Research: Feature selection and generation with small language models and web search for tabular data classification

Article Title: Selecting and generating web search based features using small language models for tabular data classification

Article References: Selecting and generating web search based features using small language models for tabular data classification. (n.d.). https://doi.org/10.1038/s41598-026-75169-w

Image Credits: AI Generated

DOI: 10.1038/s41598-026-75169-w

Keywords: Feature selection, Retrieval-augmented generation, Small language models, Tabular data, Machine learning, Open-source, Prompt design, Selecting, generating, search, based, features