The appeal of 3D data is easy to grasp. A photograph flattens the world onto a grid of pixels, discarding the very depth information that lets humans judge distance, volume and shape. Three-dimensional data, by contrast, carries precise spatial measurements, allowing an algorithm to know not just that a pedestrian is present but exactly how far away they stand and how much space they occupy. The survey identifies five dominant ways of representing this information: point clouds, voxel grids, depth and normal maps, polygonal meshes and neural fields. Each comes with trade-offs in computational cost, storage demands and expressive power, and choosing the right representation is often the first critical decision in any 3D pipeline.

Point clouds deserve particular attention because they dominate real-world sensing. Produced by LiDAR scanners and depth cameras, a point cloud is simply a set of points floating in space, each carrying a 3D coordinate and sometimes color or intensity. The trouble is that these points are irregular and unordered, which makes them fundamentally incompatible with the convolutional neural networks that revolutionized 2D image analysis. The field’s answer is sparse convolution, a clever trick that performs computation only on non-empty regions of a voxelized space. Because most of a scanned scene is empty air, sparse convolution slashes the computational burden dramatically. Its cousin, sub-manifold sparse convolution, goes further by restricting computation to kernel centers that already contain data, preventing the dilation that would otherwise destroy the input’s sparsity, though the dilation property itself can be useful for giving models contextual awareness of their surroundings.

With these foundations in place, the survey turns to the heart of modern 3D research: pre-training. The idea borrows directly from the playbook that made 2D vision so successful. Models pre-trained on ImageNet’s fourteen million labeled images learned general-purpose features that transferred effortlessly to new tasks. But 3D data lacks an equivalent of ImageNet, largely because annotating millions of 3D scans is prohibitively expensive. The solution has been self-supervised learning, in which models generate their own training signals from raw data. The survey organizes these efforts into three families: contrastive methods, masked autoencoder approaches and rendering-based techniques.

Contrastive learning teaches a network to pull similar data points together in an embedding space while pushing dissimilar ones apart, typically through an InfoNCE loss. In the 3D world this takes several forms. PointContrast, a landmark 3D-to-3D method, encourages networks to learn representations that remain stable under different viewpoints and noise, directly boosting downstream detection and segmentation. Other approaches reach across modalities: CrossPoint and SLidR bridge point clouds and rendered images, transferring knowledge from 2D backbones trained on vast image collections into 3D networks. Perhaps most strikingly, methods like PointCLIP and its successor PointCLIPV2 connect 3D perception to language, projecting point clouds into multi-view images and aligning them with the text embeddings of CLIP-style models. This enables zero-shot recognition, where a model classifies objects it has never been trained on simply by matching visual features to textual descriptions.

Masked autoencoder methods take a different route, inspired by the masked image modeling that powered recent breakthroughs in 2D vision. The core idea is to corrupt or hide parts of the input and force the model to reconstruct them, thereby learning the underlying structure of 3D shapes. VoxelMAE converts unstructured points into voxel grids and predicts the occupancy of masked voxels. PointMAE, borrowing a page from BERT, tokenizes point clouds using farthest point sampling and K-nearest-neighbor grouping, then trains with an L2 loss on masked token features; its descendant PointM2AE adds pyramid architectures that capture both fine geometric detail and high-level semantics, achieving state-of-the-art linear classification on the ModelNet40 benchmark. GD-MAE extends the paradigm to convolutional backbones favored in autonomous driving, with a masking strategy designed to prevent knowledge leakage during downsampling.

The third family is the most visually intuitive: rendering-based pre-training. Differentiable rendering makes the process of drawing an image from a 3D model mathematically transparent, so gradients can flow backward from the rendered image to the 3D representation itself. The Ponder framework describes 3D surfaces within a volume and optimizes them against 2D image projections using a NeRF-like structure, while PonderV2 scales the idea to outdoor scenes and can pre-train both 2D and 3D backbones. According to the survey, this rendering-based approach outperforms both contrastive and masked-autoencoder methods, suggesting that grounding 3D learning in the physics of image formation may be the field’s most promising direction.

These pre-trained representations then feed a rich ecosystem of downstream tasks. Classification, the field’s founding problem, began with PointNet’s pioneering use of shared multi-layer perceptrons on raw points and matured through architectures like DGCNN’s EdgeConv, which builds local graphs to learn edge embeddings. Segmentation assigns a category to every point in a scan and has spawned a fierce architectural competition: MinkowskiUNet applies U-Net-style designs to voxel data, Cylinder3D replaces cubic partitions with cylindrical ones to handle LiDAR’s varying point density, SphereFormer exploits the transformer’s global receptive field, and RangeFormer brings powerful 2D-style transformers to range-view representations. Detection, crucial for autonomous vehicles, spans LiDAR-based methods like PointRCNN and PVRCNN, camera-based systems like BEVFormer that follow Tesla’s vision-centric pipeline, and fusion approaches like PointPainting and LoGoNet that marry the precise geometry of LiDAR with the rich texture of images. Tracking, matching and registration round out the toolkit, with methods like GeoTransformer and robust variants of the classic ICP algorithm enabling machines to align separate 3D scans into coherent wholes.

Progress on all these fronts is measured against a demanding set of benchmarks. ModelNet40 and ShapeNet test synthetic shape understanding, ScanNetV2 and S3DIS probe indoor scene parsing, while KITTI, nuScenes, Waymo and ONCE push algorithms to their limits in real-world driving scenarios containing millions of annotated frames. Evaluation relies on metrics such as mean average precision, computed from intersection-over-union thresholds between predicted and ground-truth boxes, and mean intersection-over-union for segmentation, with registration tasks judged by rotation error, translation error and registration recall. The survey’s careful cataloging of these standards provides newcomers with an immediate map of how the field keeps score.

Looking forward, the authors identify several frontiers that could define the next decade. Large language models, brimming with world knowledge distilled from vast text corpora, are only beginning to be connected to 3D perception through systems like Uni3D-LLM. Knowledge transfer from the data-rich 2D and language domains into the data-starved 3D world remains a wide-open opportunity. Synthetic data could relieve the crushing cost of manual annotation, though bridging the gap between simulated and real scenes will demand advances in domain adaptation and neural rendering. Most ambitiously, the field awaits its own foundation model: while 2D vision has been reshaped by systems like Segment Anything, no equivalent exists for 3D, largely because large-scale 3D benchmarks are still lacking. Building one, the survey argues, could dramatically cut the design and deployment costs of countless 3D applications and pave the way toward a genuine era of 3D artificial general intelligence. For a field that measures the world in three dimensions, the trajectory is unmistakably upward.

Subject of Research: Self-supervised pre-training methods and downstream tasks for 3D point cloud perception

Article Title: Advances in 3D pre-training and downstream tasks: a survey

Article References: Hou, Y., Huang, X., Tang, S., He, T., & Ouyang, W. (2024). Advances in 3D pre-training and downstream tasks: a survey. Vicinagearth, 1(1), Article 6. https://doi.org/10.1007/s44336-024-00007-4

Image Credits: AI Generated

DOI: 10.1007/s44336-024-00007-4

Keywords: 3D perception, point clouds, self-supervised learning, pre-training, masked autoencoders, contrastive learning, sparse convolution, LiDAR, autonomous driving, neural rendering, foundation models, benchmarks

The survey, led by Jingren Liu of the Institute of Artificial Intelligence (TeleAI) at China Telecom together with colleagues from Tianjin University, City University of Hong Kong, and several other institutions, organizes the field around three stubborn obstacles. First is visual-token redundancy: when a video is chopped into thousands of frames, each frame is converted into hundreds or thousands of visual tokens, and the sheer volume overwhelms hardware long before any useful reasoning can happen. Second is the context-window problem: most architectures can only attend to a fixed span of tokens at once, which fragments a continuous video into disconnected snippets and destroys long-range temporal structure. Third is the reasoning gap: questions about causality, event prediction, and narrative disambiguation demand multi-step, abstract inference that goes far beyond simple perception.

The authors argue that the answer lies not in ever-larger models but in smarter inference-time engineering, and they sort existing solutions into three methodological paradigms. Selection-based methods attack redundancy directly. Systems such as VidCom² compute a distinctiveness score for each frame based on inter-frame similarity and allocate a token budget through a softmax distribution before pruning low-value tokens. METok goes further with an event-aware, multi-stage pipeline that segments continuous events by cosine similarity during encoding, prunes tokens hierarchically by attention strength during prefill, and discards low-impact entries from the key-value cache during decoding to cut both floating-point operations and memory use. DYTO achieves zero-shot compression by clustering frames into dynamic key-token groups and merging them from coarse to fine through a binary process.

Retrieval and frame-selection techniques form a second layer of the selection paradigm. APVR performs pivot frame retrieval, scores candidate frames, and then applies attention-based token filtering within the survivors, fusing spatio-temporal semantic confidence with query-aware attention. The T* framework reframes temporal retrieval as a spatial search problem, adaptively adjusting granularity across space and time to locate keyframes under extreme frame budgets. Meanwhile, adaptive keyframe sampling methods such as AKS balance relevance and coverage with a recursive judge-and-split strategy, and VSLS constructs logical triples—spatial co-occurrence, temporal proximity, attribute dependency, and causal order—to iteratively optimize which frames a model actually sees. Nar-KFC even casts keyframe selection as an integer quadratic programming problem, jointly optimizing relevance and diversity with a greedy solver while inserting coherent textual narratives to smooth temporal discontinuities.

The second paradigm replaces flat inputs with memory. Adaptive hierarchical methods such as VideoTree build a multilevel tree of key frames in real time, expanding breadth and depth according to relevance, while HEM-LLM partitions long videos into logical events and maintains multigranularity memories within and between events. Memory-augmented designs push this further: GlobalCom2 uses a global thumbnail to judge the importance of each frame’s representation and adaptively allocates compression budgets to local regions, and ∞-VIDEO borrows the idea of human sticky memory, continuously consolidating attention across a single pass through a video. For live streams, systems like LiveVLM generate and compress key-value tensors on the fly, and QuickVideo combines parallel CPU decoding, intra-group prefilling, cache pruning, and overlapping CPU-GPU execution to slash end-to-end latency.

The third and most recent paradigm treats video understanding as an active, agentic process. LVAgent deploys multiple multimodal agents in select-perceive-act-reflect cycles, collaborating across rounds without fixed sampling. VideoAgent uses a large language model as a central coordinator that iteratively identifies and compiles key information, while VCAgent adds curiosity-driven self-exploration, autonomously navigating segments to build understanding. VideoAgent2 introduces an uncertainty-aware chain-of-thought that decides when to plan, adjust, and acquire more evidence. Complementary chain-of-thought frameworks impose structure on the reasoning itself: VideoChat-A1’s chain-of-shot paradigm, which divides videos into shots and reasons across multi-round dialogues, reaches state-of-the-art accuracy of 77.0 percent on Video-MME and 70.1 percent on EgoSchema, while Self-ReS uses self-reflection and sparse attention to cut inference time by 46 percent. Retrieval-augmented pipelines such as Video-RAG and AdaVideoRAG round out the toolkit by fetching task-relevant context, with the latter adapting retrieval granularity to query complexity.

Whether these tricks actually work is the question the survey’s benchmark analysis answers, and the picture is sobering as well as encouraging. On the widely used Video-MME benchmark, the best configuration—Qwen2.5-VL-72B paired with the FlexSelect token-selection strategy—scores 74.4 overall, a modest gain over the base model’s 73.4. Larger language models clearly help: VILA* with Frame-Voyager improves from 50.5 to 60.0 when scaled from 8 billion to 34 billion parameters, and LLaVA-OneVision’s 72-billion-parameter version scores 66.3 versus 56.5 for the 7-billion variant. More input frames help too, with METok jumping from 36.4 to 46.6 on medium-length tasks when given 128 frames instead of 32. But the most striking pattern is universal: every model collapses on long videos. GPT-4o scores 71.4 on short clips yet only 55.2 on long ones, a 16.2-point gap that the authors identify as the field’s core unsolved problem.

The survey’s taxonomy of evaluation suites sharpens that diagnosis. General benchmarks such as MVBench, Perception Test, VideoVista, and Video-MME probe foundational comprehension and reveal that models score 85 to 88 percent on low-level perception but plunge to 30 to 40 percent on high-level reasoning. Hour-long stress tests like MovieChat, LVBench, MLVU, and LongVideoBench show steep degradation in contextual coherence as duration grows. Reasoning-centric benchmarks dig deeper: V-STaR forces models to produce explicit what-when-where reasoning chains, TimeLogic tests formal temporal logic, CARVE and MECD+ target causal discovery, and PhysBench spans 10,002 entries covering everything from mechanics to electromagnetism. Knowledge-grounded suites expose a particularly worrying flaw—systematic overconfidence, with models reporting confidence scores of 0.7 to 0.9 even when accuracy falls to 30 or 40 percent. Egocentric benchmarks are the harshest of all. EgoSchema shows that reliable comprehension of first-person video demands a median of 100 seconds of evidence, nearly six times the 18 seconds typical of third-person footage, and training-free systems drop from 65 to 70 percent accuracy on 30-second segments to a mere 25 to 33 percent on extended egocentric sequences.

Why pursue a paradigm with such visible weaknesses? The authors point to practical advantages that retraining cannot match. Training-free pipelines decouple capability from training budgets, adapt to new tasks without parameter updates, avoid catastrophic forgetting, and—crucially—produce auditable intermediate artifacts: the selected frames, memory traces, and reasoning chains can be inspected, so failures can be localized to specific stages such as sampling, retrieval, or reasoning. That transparency matters for the deployment scenarios the survey envisions, from professional analytics to embodied human-AI interaction and safety-critical decision making. Real-world constraints remain formidable, however: edge devices impose strict limits on compute, memory, and power; interactive applications like augmented reality demand low latency; and large-scale deployment incurs bandwidth and maintenance costs that task-oriented communication and utility-aware load shedding can only partially offset.

The survey closes with a research agenda that reads as a bridge between paradigms. Hybrid approaches could layer lightweight, parameter-efficient adaptations such as LoRA modules or mixture-of-experts on top of frozen backbones, refining only the components that need it. Adaptive memory and compression architectures should become more interpretable and resource-aware, and future benchmarks ought to measure latency, token efficiency, and long-horizon robustness alongside accuracy. Cross-modal orchestration—coordinating text, audio, and vision dynamically rather than treating video as a purely visual stream—stands out as essential for real-world generality. The overall trajectory, the authors conclude, marks a pivotal inflection: ad hoc engineering heuristics are giving way to principled, end-to-end frameworks that pair scalable computation with cognitively grounded reasoning, offering a sustainable path toward machines that can genuinely watch, remember, and reason over the endless video the modern world produces.

Subject of Research: Training-free methods, benchmarks, and open challenges for long video understanding with large multimodal models

Article Title: Towards training-free long video understanding: methods, benchmarks, and open challenges

Article References: Liu, J., Wang, Y., Zhang, L., Wang, Y., Xu, S., Wang, L., Yan, J., Zhang, D., & Chen, X. (2025). Towards training-free long video understanding: methods, benchmarks, and open challenges. Vicinagearth, 2(1), Article 6. https://doi.org/10.1007/s44336-025-00017-w

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00017-w

Keywords: long video understanding, training-free methods, vision-language models, token compression, keyframe selection, agent-based reasoning, chain-of-thought, memory architectures, benchmarks, egocentric video, multimodal AI, retrieval-augmented generation

The survey, authored by Yuankai Fan, Qizhen Weng, Yin Chen and X. Sean Wang from the Institute of Artificial Intelligence at China Telecom and Fudan University, formally defines the task as learning a translation model that maps a natural language question and a database into an executable SQL query. That definition conceals a thorny reality. Unlike typical language processing tasks with fixed input and output spaces, NL2SQL sits at the intersection of messy, unstructured human language and rigidly structured database schemas. A single question can correspond to many valid SQL translations, a one-to-many mapping that makes evaluation and training genuinely difficult. The authors argue that this intersection makes the quality, diversity and contextualization of data central to success in a way that most model-centric research has systematically underestimated.

To understand why the problem is so hard, the survey catalogs the principal technical challenges. Natural language questions suffer from lexical ambiguity, where a word like “apple” could mean a fruit or a technology company, and syntactic ambiguity, where a sentence permits multiple grammatical interpretations. Questions are also frequently under-specified: someone asking to “meet at the station” has left crucial context implicit. On the database side, modern schemas are sprawling webs of tables, columns and foreign key relationships, and queries often demand multi-table joins whose conditions must be inferred correctly. Dirty data, including missing values, duplicates and inconsistencies, compounds the risk of erroneous results. Processing entire large databases as model input is impractical, so systems must selectively compress enormous amounts of structural and content information into a usable context window.

The field has traveled a long road to reach its current state. Early systems relied on hand-crafted rules and templates, which produced syntactically correct queries but collapsed when confronted with linguistically complex questions involving nested clauses, coreference or ellipsis, and required laborious manual updates for every new domain. Deep learning then brought sequence-to-sequence encoder-decoder models, with systems such as SQLNet framing translation as a slot-filling problem, TypeSQL injecting type information from knowledge graphs, and IRNet introducing intermediate representations that abstract away from raw SQL. Pre-trained language models like BERT and RoBERTa later raised accuracy further, though they still stumbled on outer joins and aggregations and degraded sharply across domains. The arrival of large language models, from the GPT series onward, has powered the current generation of systems, using prompt engineering for proprietary models or fine-tuning for open ones, with approaches such as DIN-SQL, DAIL-SQL and MAC-SQL achieving leading results.

The intellectual core of the survey is its taxonomy of five data types that together define the NL2SQL lifecycle. External knowledge includes domain-specific ontologies, knowledge graphs and common-sense resources, as well as knowledge derived from large language models themselves, which can fill gaps the database does not cover. Text corpora come in two flavors: annotated datasets pairing natural language questions with gold SQL queries, and raw unannotated text such as user query logs and documentation that supports pretraining. Database schemas provide the structural blueprint of tables, columns, data types and relationships, serving as the critical reference for mapping language onto the database. Database instances, the actual rows of stored records, ground the semantics and help resolve ambiguity. Finally, execution feedback, comprising query results and error messages, closes the loop by enabling error correction, verification and reinforcement learning.

These data types map onto four stages of the pipeline. During query understanding, external knowledge helps recognize user intent and entities, compensating for missing or implicit semantics in the input. Schema linking follows, identifying the relevant tables, columns and cell values, using classifiers, graph neural networks or prompting methods, with metadata such as column descriptions and foreign key constraints providing valuable semantic signals. SQL generation then treats the model as a translator, typically fed schema and instance data, often with few-shot examples for LLM-based systems, and supported by fine-tuning on large-scale text corpora. The final stage, query post-refinement, is where the authors see the greatest untapped potential. Most current methods lean almost exclusively on execution feedback, checking whether a query runs and returns plausible results, but this single signal cannot capture subtle semantic inconsistencies in query logic.

The survey also traces how benchmarks have evolved to stress-test these systems. Early datasets such as ATIS for flight information and Geo for United States geography were single-domain with simple queries. The field then shifted to cross-domain evaluation, led by WikiSQL, drawn from Wikipedia tables, and Spider, which introduced complex multi-table SQL across many domains along with extensions like Sparc and CoSql for contextual and conversational settings. Most recently, large-scale real-world benchmarks including BIRD, ScienceBenchmark and Spider 2.0 feature naturally occurring questions, authentic enterprise schemas and challenging structures such as nested queries and set operations, testing reasoning, schema linking and robustness to ambiguity in ways that matter for deployment.

Measuring success is itself a nuanced science. Exact match accuracy checks whether generated SQL is literally identical to the ground truth, but underestimates performance because the same intent can be expressed in many syntactically different ways. Execution accuracy compares the results of running the generated query against those of the reference query, yet risks overestimating correctness when different logic happens to produce identical outputs. Test-suite accuracy executes predictions on curated sets of randomly generated databases to probe semantic equivalence more rigorously, while the valid efficiency score adds a crucial practical dimension by measuring how efficiently correct queries run. In their comparative analysis of thirteen state-of-the-art systems on Spider and BIRD, the authors observe that LLM-based methods substantially outperform their pre-trained predecessors, and notably show stronger generalization on test sets than development sets, suggesting genuine robustness rather than benchmark overfitting.

Looking forward, the authors lay out a research agenda squarely centered on data. They propose dedicated query-rewriting modules that clarify ambiguous or context-dependent expressions before schema linking begins, borrowing proven techniques from dialogue systems to reduce error propagation. They call for richer post-refinement that integrates user interaction signals, semantic validation against schema constraints and cross-checking with alternative query formulations, rather than relying on execution feedback alone. Bridging NL2SQL with the broader natural language to code field could import multi-step reasoning, program synthesis, formal verification and human-in-the-loop debugging. Even structured noise, drawing on recent positive-incentive noise research, might be injected to improve resilience against ambiguous inputs. Beyond accuracy, real deployment demands efficiency and dialect compatibility, suggesting systems should exploit metadata such as indexes and statistics like column cardinality to generate queries optimized for the target engine. The message of the survey is ultimately a liberating one: the fastest path to databases anyone can talk to may run not through bigger models, but through smarter data.

Subject of Research: A data-centric survey of natural language to SQL translation systems, covering data types, benchmarks, evaluation metrics and future research directions.

Article Title: Rethinking data in NL2SQL: a survey of what we have and what we expect

Article References: Fan, Y., Weng, Q., Chen, Y., & Wang, X. S. (2025). Rethinking data in NL2SQL: a survey of what we have and what we expect. Vicinagearth, 2(1), Article 15. https://doi.org/10.1007/s44336-025-00026-9

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00026-9

Keywords: NL2SQL, Text-to-SQL, large language models, databases, SQL generation, benchmarks, data-centric AI, schema linking, pre-trained language models, evaluation metrics, natural language processing, database interfaces

The survey, led by Jinyang Chen, Haolun Wu, Jianhong Pang, and Yihua Wang with senior supervision from Dell Zhang and Changzhi Sun, provides the most systematic account to date of what the field calls tool learning. The core idea is deceptively simple: instead of expecting a language model to answer every question from its internal parameters, the model should learn to delegate parts of a problem to external tools that are faster, more accurate, or more current. When a user asks about tomorrow’s weather in Tokyo, the model should not hallucinate a forecast but call a live weather API. When a calculation involves more than trivial arithmetic, the model should hand the numbers to a code interpreter or symbolic solver. This reframing, the authors argue, turns tool use from physical manipulation into symbolic orchestration, where the language model acts as an intelligent controller that understands both what the user wants and what each tool can do.

At the heart of the survey lies a unified four-stage framework that the researchers use to organize the entire landscape: task planning, tool selection, task execution, and response generation. In the planning stage, the model decomposes a complex, high-level instruction into a series of smaller, solvable subtasks, working out the dependencies and the order in which they must run. Systems like ART build libraries of example tasks that guide this decomposition through few-shot prompts, while HuggingGPT combines specification-based instructions with demonstration parsing to schedule subtasks, and RestGPT refines its plans iteratively in a coarse-to-fine scheme as execution proceeds. Planning, the authors stress, is the cognitive foundation on which everything else depends, because a model that cannot break a problem down correctly will select the wrong tools or call them in the wrong order.

Tool selection is the second stage, and it presents a genuinely difficult engineering problem when the number of candidate tools reaches into the thousands. The survey distinguishes two dominant strategies. Retriever-based approaches first filter the tool library using semantic relevance, drawing on classical term-matching techniques such as TF-IDF and BM25 or on neural models like Sentence-BERT trained specifically for tool retrieval. Newer systems push this further: Craft asks the language model to generate hypothetical tool descriptions from a query and retrieve against them, while COLT applies graph neural networks and explicitly targets completeness of retrieval, ensuring no relevant tool is missed. When the candidate pool is small, LLM-based selection takes over, letting the model reason directly over tool descriptions. Methods like ToolBench combine fine-tuning with example retrieval and system prompts, while ToolVerifier generates comparative questions that force the model to distinguish between superficially similar tools, reducing misselection in ambiguous contexts.

Task execution, the third stage, is where natural language must be converted into structured, machine-readable calls. The survey illustrates this with a simple but revealing example: to convert 100 dollars into euros, the model must recognize that a currency API is required, extract the amount, source currency, and target currency, and emit a well-formed call such as convert with the appropriate named parameters. Getting this right demands schema-guided prompting, as in RestGPT and EasyTool, or more elaborate reasoning strategies like ReverseChain, which works backward from the desired final tool to infer the intermediate steps and parameters. Multi-agent architectures such as ToolNet and ConAgents divide the labor among specialized sub-agents that parse, plan, and execute, negotiating with each other over a shared backbone model. Meanwhile, instruction-tuned systems like Gorilla and ToolkenGPT, trained on massive tool-use corpora, generalize to unseen APIs with remarkable fluency, and verification modules such as ToolVerifier and Themis check proposed calls for validity before anything is actually executed, catching unsafe or ill-formed invocations early.

The final stage, response generation, determines how tool outputs are woven back into the model’s answer. The survey identifies two broad paradigms. Direct insertion methods, exemplified by TALM and Toolformer, embed the raw tool output at placeholder positions in the prompt and let the language model continue from there; Toolformer went further by fine-tuning on synthetic data in which API calls are interleaved with text, teaching the model when and where to invoke tools during generation. Information integration methods go deeper: ToolLLM routes tool outputs through a dedicated integration module before the main model composes its answer, ReCOMP compresses retrieved and computed information into latent representations for tighter factual grounding, and ConAgents lets multiple agents reinterpret tool results collaboratively before drafting the final response. The trade-off is clear: direct insertion is fast and simple, while deeper integration produces more coherent, context-sensitive answers in multi-tool scenarios.

Perhaps the most consequential part of the survey is its method-centric synthesis of how these capabilities are actually learned. Tuning-free approaches rely purely on prompt engineering and in-context demonstration, making them ideal for proprietary models that cannot be retrained and for rapid prototyping against unseen APIs; ReAct, which interleaves chain-of-thought reasoning with tool calls and incorporates environmental feedback, remains the canonical example. Supervised fine-tuning trains models on curated traces of tool interaction, from Toolformer’s self-generated annotations to ToolBench’s multi-stage datasets, with newer methods such as SFT-GO optimizing semantically important tokens separately and rehearsal-based strategies preventing catastrophic forgetting of general abilities. Reinforcement learning, however, is emerging as the frontier. ToolRL showed that fine-grained, step-level rewards outperform coarse final-answer signals and achieved up to a 17 percent improvement over supervised fine-tuning using Group Relative Policy Optimization. OTC penalizes unnecessary tool calls and cut tool usage by up to 73 percent while boosting productivity by 229 percent, and ReTool, which combines code execution with textual reasoning, reached 72.5 percent accuracy on the AIME mathematics competition, surpassing even OpenAI’s o1-preview baseline while exhibiting emergent self-correction.

Evaluation has matured alongside the methods. The survey reviews a rich ecosystem of benchmarks, from broad, coverage-oriented suites like ToolBench, API-Bank, and APIBench that test generalization across hundreds or thousands of APIs, to diagnostic instruments such as T-Eval, which scores six distinct dimensions of tool competence, and the Berkeley Function Calling Leaderboard, which measures structured invocation and agentic orchestration. Scenario-specific benchmarks probe the edges: ToolQA and ToolTalk test tool-grounded question answering in dialogue, ToolEmu simulates tools safely when real execution is too costly or risky, InjecAgent probes robustness against adversarial injection attacks, and SCITOOLBENCH and RoTBench examine scientific and symbolic tool chaining. The reported results paint a consistent picture. GPT-4-Turbo leads T-Eval with an overall score of 86.4, and the GPT-4 family dominates the GTA benchmark, yet the open-source xLAM series tops single-turn function calling on BFCL with accuracy up to 89.27, and the fine-tuned Lynx-7B model approaches GPT-3.5 performance on API-Bank, demonstrating that high-quality, ability-diverse training data can narrow the gap considerably.

The stakes extend well beyond leaderboards. The authors argue that tool learning makes language models more trustworthy and interpretable in concrete ways: intermediate tool invocations expose the reasoning path, standardized tool interfaces reduce sensitivity to prompt phrasing, and deterministic, externally verifiable tool outputs help suppress the hallucinations that have plagued large language models since their inception. In high-stakes domains such as finance, law, and healthcare, this traceability is not a luxury but a requirement. Tool integration also lets models engage with databases, scientific solvers, and medical systems, producing results that are domain-specific and checkable rather than merely plausible. This transparency and grounding, the survey contends, is what elevates tool learning from a convenient trick to a foundational capability that redefines what machine intelligence can credibly deliver.

Significant open challenges remain, and the survey is candid about them. Safety is paramount: hallucinated API calls or erroneous parameter choices in open-ended environments can produce dangerous or misleading outcomes, demanding runtime checks, input sanitization, and robust fallback strategies. Latency bottlenecks from multi-step, multi-tool pipelines strain user experience and scalability. Seamless multimodal integration across vision, speech, and structured data requires better interface design, and personalization, incorporating user preferences and history into tool selection, is still largely unsolved. The authors point toward multi-agent collaboration, LLM-driven tool creation in which models synthesize their own new functions, and unified abstraction frameworks that standardize model-tool interaction while enhancing generalization and safety. Richer benchmarks that simulate authentic, multi-stage, multimodal task scenarios are needed to capture the complexities of real-world use. If those challenges are met, the researchers conclude, tool learning will not merely augment language models but will stand as a foundational component of autonomous, transparent, and reliable artificial intelligence, marking the moment machines truly began to extend their own capabilities the way humans have extended theirs for millions of years.

Subject of Research: Tool learning with large language models, covering methods, pipelines, tuning strategies, and benchmarks for tool-augmented AI systems.

Article Title: Tool learning with language models: a comprehensive survey of methods, pipelines, and benchmarks

Article References: Tool learning with language models: a comprehensive survey of methods, pipelines, and benchmarks. (n.d.). https://doi.org/10.1007/s44336-025-00024-x

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00024-x

Keywords: tool learning, large language models, artificial intelligence, reinforcement learning, supervised fine-tuning, API integration, benchmarks, task planning, hallucination, AI agents, natural language processing, foundation models

The motivation is straightforward. LLMs are increasingly deployed in high-risk sectors such as healthcare, finance and law, where a confident but wrong answer can have serious consequences. Yet the internal decision processes of these models remain notoriously opaque. The survey catalogues the familiar litany of failures: hallucinations, brittleness under distribution shift, bias, security and privacy risks, and limited interpretability. The authors argue that expecting full transparency from a purely transformer-based model may be unrealistic, and that symbolic components, which can provide explicit structure, logical constraints and reasoning support, offer a promising complement rather than a replacement.

Neurosymbolic AI is not new. Frameworks such as Logic Tensor Networks, DeepProbLog, Neural Logic Machines and the Neural Theorem Prover were developed for conventional neural networks, combining differentiable learning with logical inference. But the Hull team contends that these frameworks do not transfer cleanly to LLMs. Language models differ fundamentally from the neural architectures for which earlier neurosymbolic methods were designed: they operate at enormous parameter scale, generate text autoregressively token by token, and produce context-dependent outputs. End-to-end joint training of symbolic and neural components, a hallmark of classical neurosymbolic systems, becomes expensive or impractical at LLM scale. Instead, integration is typically achieved through fine-tuning, prompt engineering or external knowledge injection.

To bring order to a sprawling literature, the survey proposes a novel taxonomy organised along four dimensions. The first is the stage of the LLM lifecycle at which symbolic information enters: pre-training, training, fine-tuning, or inference. The second is the coupling mechanism, ranging from decoupled designs in which the LLM and symbolic engine operate autonomously, to intertwined architectures in which symbolic structure directly shapes hidden states or even the training objective. The third dimension distinguishes algorithm-level integration, where symbolic knowledge is embedded within the model’s architecture and representations, from application-level integration, where external symbolic resources are connected through workflows such as retrieval and verification. The fourth distinguishes the architectural paradigms through which the two worlds communicate.

Those paradigms form perhaps the survey’s most useful contribution. In the LLM-to-Symbolic pipeline, the language model translates natural language into formal structures that a symbolic engine can then execute or verify. Systems such as LINC and Logic-LM convert problems into first-order logic and hand them to theorem provers or satisfiability solvers, while Symbolic Chain-of-Thought uses logic rules to guide step-by-step reasoning. In the opposite direction, the Symbolic-to-LLM pipeline injects structured knowledge from knowledge graphs, ontologies or logic engines into the model, often through retrieval-augmented generation. Approaches such as RoG, or Reasoning on Graphs, train models to follow knowledge-graph relation paths as explicit reasoning plans, producing answers that are both grounded and traceable. Hybrid models combine both directions in bidirectional, iterative architectures, exemplified by the LLM-Modulo framework, in which model-based verifiers critique and refine LLM-generated plans.

The quantitative picture that emerges from the taxonomy is telling. Most existing work clusters around inference-stage integration, moderate or loose coupling, and application-level designs. The authors interpret this concentration as a pragmatic preference for modularity: bolting symbolic modules onto a frozen LLM avoids costly retraining and deep architectural surgery. Relatively few studies attempt tight coupling, such as loss-level integration in which symbolic constraints are folded directly into the optimisation objective, as KEPLER does by jointly optimising masked language modelling with a knowledge-graph embedding loss. That gap, the survey suggests, represents both an engineering challenge and an opportunity for deeper alignment between symbolic and neural reasoning.

Evaluation receives equally critical treatment. The review catalogues the benchmarks used to assess knowledge-graph-integrated and logic-integrated LLMs, from GLUE, SuperGLUE and CommonsenseQA to FOLIO, ProofWriter, LogicBench and Multi-LogiEval, alongside domain-specific suites for mathematics, coding and physics such as GSM-Symbolic, CodeXGLUE and ARB. But the authors are blunt about the shortcomings. Standard metrics like accuracy, F1 and BLEU capture only final-answer correctness or lexical overlap; they cannot distinguish genuine logical reasoning from surface pattern matching, nor can they separate the contribution of the symbolic component from the LLM’s own parametric knowledge. Data contamination, limited coverage of reasoning modes, and the incompleteness of knowledge graphs further muddy the waters. The survey recommends process-level evaluation that scores intermediate reasoning steps, symbolic consistency metrics that test formal entailment, and ablation-based attribution that isolates what symbolic grounding actually adds.

On the application side, the review documents how symbolic integration is already sharpening LLM capabilities. Knowledge-enhanced embeddings and adapters, from K-BERT and ERNIE to KnowBert and LambdaKG, enrich representations with structured facts. Reasoning frameworks combine LLM-generated intermediate steps with symbolic verification, with LLM-ARC, which pairs a language model with an Answer Set Programming critic, reaching 88.32 percent accuracy on the FOLIO benchmark. Planning systems such as LLM-Planner, Plansformer and expert-free LLM-symbolic pipelines generate executable action schemas from natural language. The authors also propose a three-way taxonomy of hallucination origins in hybrid systems: parametric hallucinations arising from the model’s own weights, symbolic hallucinations from outdated or inconsistent knowledge graphs, and integration hallucinations born at the neural-symbolic interface itself, each demanding different remedies.

The survey closes with a sober assessment of what remains unsolved. Design patterns for LLM-symbolic integration lack systematic formalisation; conflict resolution between symbolic modules and model outputs remains ad hoc; knowledge editing risks cascading side effects; and graph linearisation and computational overhead still hamper efficient integration. Tightly coupled and compiled forms of integration remain comparatively underexplored, and many published systems remain conceptual or benchmark-scale. Yet the direction of travel is clear. As LLMs continue to improve through chain-of-thought prompting and tool-augmented inference, the authors argue, symbolic integration should be understood not as a rival but as a means of grounding increasingly capable models in verifiable, explainable knowledge, precisely the assurance that high-stakes domains demand. For a field racing to make artificial intelligence trustworthy, this roadmap may prove one of its most important signposts.

Subject of Research: Integration of symbolic AI techniques such as knowledge graphs and logic into large language models to enhance reasoning and explainability

Article Title: Neurosymbolic Large Language Models: A Survey of Symbolic Integration, Reasoning and Explainability

Article References: Rani, M., Mishra, B. K., & Thakker, D. (2026). Neurosymbolic Large Language Models: A Survey of Symbolic Integration, Reasoning and Explainability. Information Systems Frontiers. https://doi.org/10.1007/s10796-026-10794-4

Image Credits: AI Generated

DOI: 10.1007/s10796-026-10794-4

Keywords: Neurosymbolic AI, Large language models, Symbolic integration, Knowledge graphs, Reasoning, Explainability, Logic integration, Retrieval-augmented generation, Benchmarks, Hallucination, Trustworthy AI, Systematic review