Researchers have unveiled a new artificial intelligence framework designed to make complex database searches dramatically faster, combining large language models with adaptive query optimization inside key-value storage systems. Called LLM-Enhanced Adaptive Pathway Optimization, or LAPO, the system targets a problem that has quietly become central to modern computing: how to retrieve useful information quickly when databases contain enormous volumes of data, complicated relationships, and multiple layers of storage. The work, published in the Journal of Big Data, focuses on MyRocks, a database engine built on the Log-Structured Merge-tree architecture. According to the researchers, LAPO can reduce execution time and latency for demanding join queries by changing not only how a query is written, but also how the database chooses its internal route through stored data.
The challenge begins with the architecture of many high-performance key-value stores. Unlike conventional relational databases, which organize information into tables and often rely on sophisticated indexing structures, key-value systems store data as pairs consisting of a unique key and an associated value. This design is highly efficient for rapid writes and large-scale data ingestion, making it attractive for cloud services, web platforms, recommendation systems, and other applications that must continuously absorb information. Yet the same design can become difficult to navigate when users ask complex questions involving joins, nested subqueries, filtering, sorting, and relationships spread across multiple data structures. MyRocks uses an LSM-tree, in which newly written records are first collected in memory and then progressively merged into files across several storage levels. A read may therefore require checking multiple layers, amplifying the cost of a poorly chosen execution plan.
In a conventional database optimizer, a query is translated into a collection of possible execution strategies. The system estimates the cost of each strategy and selects the one it believes will be fastest. That process works well for many routine workloads, but it can struggle when the number of possible query paths becomes very large or when the optimizer’s cost estimates fail to capture the behavior of an underlying key-value store. A plan that looks efficient on paper may force the database to examine many LSM-tree levels, perform unnecessary key lookups, or execute joins in an unfavorable order. The result can be a local optimum: a plan that is better than the alternatives the optimizer considered, but far from the best plan that could have been found with a wider search. LAPO was developed to address this limitation by broadening the search and using machine learning to identify promising routes.
The framework begins with a method the researchers call multi-branch path matching. During query parsing, this technique generates several possible parameter configurations rather than committing immediately to one set of optimizer choices. These parameters may influence such decisions as join ordering, access paths, filtering behavior, and the way operations are pushed through the query plan. By creating multiple branches, LAPO expands the optimizer’s search space and increases the chance of discovering a plan that conventional rule-based selection would overlook. The approach is intended to preserve the speed of automated optimization while avoiding the narrow decision-making that can occur when a system follows only its first plausible route. In practical terms, the database is encouraged to consider several competing interpretations of how the query should travel through the storage engine.
The second component, known as dependency chain regulation, attempts to explain how a user’s SQL logic is connected to the low-level operations performed by the key-value store. A complex query is not a single action. It may trigger scans, point lookups, range searches, table combinations, temporary results, and repeated access to multiple storage layers. LAPO models the relationships among query parameters, execution paths, and underlying operations in what the researchers describe as a parameter-path-operation association matrix. Candidates that receive high scores in this matrix are retained for deeper evaluation, while less promising combinations are removed earlier. This filtering is important because exhaustive exploration can itself become a performance problem. Instead of testing every possible configuration, LAPO tries to spend computational effort only on pathways that appear structurally compatible with the query and the storage engine.
LAPO then adds an adaptive parameter adjustment model that uses features of the query path as input to a pre-trained neural network. These features can represent characteristics such as the complexity of joins, the expected distribution of records, the number of storage levels likely to be accessed, and the relationship between logical operations and physical reads. The model predicts which execution-plan parameters are most likely to produce efficient behavior, allowing the optimizer to adjust its decisions dynamically rather than relying entirely on fixed settings. This is a significant shift from traditional database tuning, where administrators often configure parameters manually or use static rules based on historical workloads. A learned model can, in principle, respond to changing data distributions and query patterns, although its success depends on the quality and representativeness of the data used to train and evaluate it.
The final layer, called Unnesting SQL Optimization, uses a fine-tuned large language model to rewrite certain nested SQL queries into equivalent unnested forms. Nested queries can be intuitive for people to write, but they may cause a database to repeat calculations or evaluate intermediate results more times than necessary. Unnesting transforms the logical structure so that operations can be combined, reordered, or processed more directly. The LLM is not presented as an autonomous database administrator making unrestricted changes. Instead, it is used as an external transformation mechanism focused on recognizing complex nested patterns and generating a simpler representation that can be passed to the optimizer. By reducing redundant computation before physical execution begins, this stage complements the pathway search and parameter prediction performed by the other parts of LAPO.
The researchers evaluated the framework using three real-world datasets: IMDB, which represents interconnected information about films and people; Chinook, a compact commercial-style database containing music and customer records; and OpenFlights, which includes data related to airports, airlines, and routes. These datasets were used to test complex join workloads and compare LAPO with native MyRocks behavior. The reported results show that LAPO substantially improved query efficiency, particularly in scenarios where multiple joins and complicated query structures placed heavy demands on the storage engine. The gains came from the combined effect of broader plan exploration, early elimination of ineffective candidates, adaptive parameter selection, and SQL rewriting. Rather than attributing the improvement to a single machine-learning prediction, the study presents performance as the result of a multi-phase optimization pipeline.
The work arrives as database systems increasingly face workloads that were never anticipated by older optimization techniques. Modern applications generate data continuously, distribute it across heterogeneous hardware, and expect near-instant answers to questions that may involve millions or billions of records. Key-value stores are excellent at scaling write-heavy operations, but complex analytical queries can expose weaknesses in their read paths. LAPO’s significance lies in treating query optimization as a coordinated interaction among language understanding, search, machine learning, and storage-engine behavior. The approach also highlights the growing role of LLMs beyond chatbots and text generation: in this case, the model is used to interpret and restructure formal database instructions, while specialized optimization components determine how those instructions should be executed.
There are still important questions about how LAPO would perform beyond the study’s datasets and experimental conditions. A learned optimizer must remain reliable when data distributions change, new query types appear, or the underlying hardware differs from the environment used during training. SQL rewrites must also preserve semantics, because an apparently simpler query can produce incorrect results if its transformations do not respect null values, duplicate rows, aggregation rules, or ordering requirements. The researchers describe LAPO as an extension of earlier work called Adaptive Pathway Optimization, presented at the APWeb-WAIM 2025 conference, with the new study adding the LLM-based unnesting layer and revising the broader optimization framework. If the method proves robust across more database engines and production-scale workloads, it could point toward a new generation of self-tuning data systems—ones that do not merely store information at high speed, but continuously learn the most efficient route to retrieve it.
Subject of Research: Artificial intelligence-assisted query optimization for LSM-tree-based key-value stores
Article Title: LAPO: LLM-enhanced adaptive pathway optimization for complex query tasks on key-value stores
Article References: Chai, Y., Li, X., Zhang, Q. et al. “LAPO: LLM-enhanced adaptive pathway optimization for complex query tasks on key-value stores.” Journal of Big Data (2026).
Image Credits: AI Generated
DOI: 10.1186/s40537-026-01552-z
Keywords: Query optimization, reduce searching cost, LSM-tree, LLM
