From ELIZA to RAG: How Chatbots Learned to Look Up Facts Before They Speak

Chatbots have come a long way from the pattern-matching scripts of the 1960s, but the field’s newest and most consequential shift is only now being mapped in detail. A comprehensive survey published in Discover Artificial Intelligence by researchers at Symbiosis International University in Pune traces the full arc of conversational AI, from rule-based systems like ELIZA through purely generative large language models, and arrives at the architecture that now dominates serious deployments: Retrieval-Augmented Generation, or RAG. Rather than cataloguing techniques chronologically, the team systematically coded 106 publications across eight design dimensions, arguing that RAG chatbots are not monolithic models but configurable pipelines whose reliability depends on how components interact. That reframing, the authors contend, is what the field needs to move from fluent demos to trustworthy systems.

The historical starting point is instructive. ELIZA, written by Joseph Weizenbaum in 1966, sustained engaging conversations using nothing more than string matching and templated responses, famously simulating a Rogerian psychotherapist by rephrasing user statements as questions. Finite-state dialogue systems and frame-based architectures with slot-filling mechanisms followed, powering predictable, task-oriented applications such as interactive voice response. These systems were interpretable and controllable, but their rigidity, poor scalability, and inability to acquire new knowledge made them fundamentally unsuited to open-ended dialogue. The survey treats them as essential context: they established the principles of human-computer conversation while exposing the ceiling of hand-crafted rules.

Generative chatbots built on Transformer architectures shattered that ceiling, producing fluent, context-aware responses across virtually any topic. Yet the survey is blunt about their structural weaknesses. Because factual knowledge lives implicitly in model weights, generative systems hallucinate, especially on ambiguous or out-of-distribution queries; their knowledge freezes at training time; and their outputs lack transparent attribution, a serious problem in regulated domains like healthcare, finance, and law. Even high average benchmark performance conceals occasional but confident failures that erode user trust. Scale alone, the authors argue, has proven insufficient to guarantee reliability in real-world conversational settings.

RAG emerged as the answer by decoupling knowledge access from model parameters. Instead of relying solely on memorized facts, a RAG system retrieves relevant documents at inference time and conditions its response on that evidence. The survey models this as a modular pipeline: query processing, knowledge chunking and representation, retrieval, optional re-ranking, retrieval-generation fusion, response generation, and finally grounding, attribution, and safety controls. Design decisions at earlier stages propagate downstream, which is why the authors insist that RAG must be understood as a family of architectures with internal trade-offs rather than a single algorithm. Their framework identifies six core dimensions, from chunking strategy to safety controls, that recur across the literature.

Those trade-offs are quantitatively striking. Shrinking chunk size from 512 to 128 tokens improves recall@5 by roughly 8 to 12 percent on open-domain benchmarks, but degrades generation coherence by 3 to 5 percent because fragmented context forces the generator to reconstruct meaning, elevating hallucination risk. Cross-encoder re-ranking can lift precision by up to 15 percent in mean reciprocal rank, yet adds 200 to 300 milliseconds of latency per query, a cost that can violate real-time interaction constraints. Dense retrieval cuts retrieval latency by 40 to 60 percent compared with lexical methods at scale, but demands costly vector indexing infrastructure. Hybrid pipelines that combine sparse lexical retrieval with dense semantic matching are increasingly favoured in production, balancing efficiency against robustness at the price of orchestration complexity.

The survey’s failure-mode analysis may be its most valuable contribution. RAG chatbots fail in distinctive, systematic ways: retrieval returns semantically similar but pragmatically irrelevant passages that anchor responses to wrong assumptions; heterogeneous sources yield contradictory evidence that generators resolve arbitrarily; retrieved text can carry prompt injections that override system intent; and generators engage in citation laundering, attaching plausible-looking references to unsupported claims. Perhaps most troubling, models with high BERTScore values still produced unsupported claims in 28 to 35 percent of responses when evaluated against their retrieved evidence. The authors also highlight a subtler insight from recent in-context learning research: retrieved passages act as demonstrations whose effectiveness is model-dependent, so optimal retrieval must account for the specific generator’s conditional entropy, not just generic similarity.

Evaluation practices come in for sharp criticism. Standard metrics like BLEU and ROUGE correlate only weakly with human judgments of factual consistency, with Pearson coefficients of roughly 0.21 to 0.38 across knowledge-grounded dialogue benchmarks. Systems with identical retrieval performance produced generation quality varying by more than 15 percentage points in human-rated faithfulness depending on fusion strategy, showing that end-task metrics conflate retrieval quality with generation quality. Newer frameworks attempt to fix this: RAGAS decomposes quality into faithfulness, answer relevance, and context relevance; ARES adds statistically rigorous confidence intervals via prediction-powered inference; and RAGChecker diagnoses whether errors stem from the retriever or the generator. LLM-as-judge approaches scale well but inherit self-preference bias, inflating scores by 5 to 15 percent when evaluator and generator share a model family, plus position bias tied to passage ordering.

Domain case studies show there is no universal RAG configuration. In healthcare, MedRAG retrieved from five million PubMed abstracts to reach 78.3 percent accuracy on the MedQA-USMLE benchmark, against 60.2 percent for non-RAG GPT-4, while systems with explicit grounding achieved clinician-rated accuracy of 86 to 92 percent versus 68 to 74 percent without it. Legal deployments demand citation accuracy above 95 percent and jurisdictional tagging, with hybrid retrieval reaching an MRR of 0.87 but incurring 350 milliseconds of extra latency. Enterprise systems such as FinRAG process over 10,000 queries daily at 1.8-second average latency while maintaining fine-grained access controls, and educational platforms report 82 percent student satisfaction with RAG-based tutoring versus 65 percent for non-RAG baselines. Regulatory frameworks from HIPAA to the EU AI Act increasingly make grounding verification a compliance requirement rather than an optional feature.

Looking forward, the survey identifies frontiers that could define the next decade of conversational AI. Self-RAG teaches a single model to decide when retrieval is needed and critique its own outputs using reflection tokens, while Corrective RAG deploys a lightweight evaluator that discards poor retrievals and falls back to web search. Agentic RAG treats retrieval as a tool invoked on demand within reasoning loops, and GraphRAG exploits knowledge graphs for multihop reasoning that flat retrieval cannot capture. The authors call for controlled ablation experiments that isolate chunking, retrieval depth, and fusion effects under fixed generator conditions, alongside privacy-preserving retrieval pipelines, attribution-aware generation, and multimodal grounding. Their central message is that RAG chatbots are evolving socio-technical systems: building ones that are genuinely reliable requires managing interactions across the whole pipeline, not optimizing components in isolation.

Subject of Research: Retrieval-augmented generation architectures for knowledge-grounded conversational AI systems

Article Title: A Survey of conversational AI from rule based to generative and retrieval augmented generation chatbots

Article References: Ghaywat, V., Singh, A., Shahade, A. K., Khan, W., & Deshmukh, P. V. (2026). A Survey of conversational AI from rule based to generative and retrieval augmented generation chatbots. Discover Artificial Intelligence, 6(1), Article 1292. https://doi.org/10.1007/s44163-026-02373-y

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02373-y

Keywords: retrieval-augmented generation, chatbots, conversational AI, large language models, hallucination, dense retrieval, knowledge grounding, Self-RAG, GraphRAG, evaluation metrics, healthcare AI, systematic survey