For decades, machines have been trained to recognize speech, classify environmental sounds and analyze music as separate technical problems. A new review argues that this fragmented approach is giving way to a broader ambition: building machines with something closer to general auditory intelligence. Instead of treating audio as a narrow stream of acoustic signals, researchers are increasingly combining sound-processing systems with large language models capable of reasoning, describing events, generating responses and interacting with people. The goal is not merely to identify a siren or transcribe a sentence, but to understand what is happening, why it matters and how a machine should respond.
The shift is being driven by the unique information carried through sound. Audio can reveal language, identity, emotion, location, physical activity and social context, often when visual information is unavailable. A voice can communicate uncertainty or excitement through pitch, rhythm and intensity, while background sounds can indicate whether a person is walking through a crowded station, working in a kitchen or approaching a dangerous environment. Unlike a still image, sound also unfolds over time, requiring machines to track sequences, changes and relationships between events. The review, published in Nature Machine Intelligence, examines how recent advances are bringing these capabilities into large language model-based systems.
At the center of this transformation are models that connect audio representations to the language-based reasoning abilities of large language models. Raw sound waves are usually converted into compact mathematical representations by an audio encoder, often using techniques related to spectrogram analysis. A spectrogram maps frequencies over time, making it possible for neural networks to detect patterns associated with speech, musical structure or environmental events. These representations can then be aligned with tokens or embeddings processed by a language model. Once the connection is established, the system can answer questions about a recording, explain an acoustic event, summarize a conversation or reason across several sounds rather than simply attaching a label to one clip.
This approach is expanding audio comprehension beyond conventional recognition tasks. Earlier systems might have been designed to determine whether a recording contained a dog bark, a car horn or a spoken command. Language-model-based systems can potentially describe interactions among multiple sounds, infer the context of an event and respond to natural follow-up questions. A user might ask what changed during a recording, which speaker sounded distressed or whether a warning signal occurred before a mechanical failure. Such questions require temporal reasoning, acoustic discrimination and contextual interpretation. They also expose a major challenge: models must learn not only what sounds resemble, but what those sounds mean in real-world situations.
Large language models are also reshaping audio generation. Traditional speech synthesis systems generally converted text into speech with a predetermined voice and limited control over delivery. Newer systems aim to generate speech that reflects conversational context, emotion, emphasis and individual speaking style. The same broader framework can be extended to music, sound effects and environmental audio. In principle, a model could create a spoken explanation, a realistic background scene or a coordinated mixture of voices and sounds from a textual instruction. The technical difficulty lies in maintaining timing, coherence and expressive detail. Audio unfolds continuously, so a generated output must remain consistent from one moment to the next rather than merely producing plausible isolated fragments.
The review identifies speech-based interaction as one of the most visible pathways toward human-like machine behavior. Voice communication is faster and more natural than typing for many situations, but convincing spoken interaction requires more than accurate transcription. A responsive system must detect when a person begins and ends speaking, recognize interruptions, interpret hesitation and understand conversational intent. It must then generate an answer quickly enough to preserve the rhythm of dialogue. Speech-to-speech systems seek to reduce the delay and information loss that can occur when spoken input is first converted into text and later synthesized back into audio. Preserving tone, timing and emotion could make interactions feel less like exchanges with a software interface and more like conversations with an attentive partner.
That promise comes with demanding engineering constraints. Real environments contain reverberation, overlapping speakers, traffic, machinery and unexpected interruptions. Microphones may capture only partial or distorted signals, while speakers may use slang, code-switch between languages or express meaning indirectly through tone. A model that performs well on clean laboratory recordings can fail when conditions become noisy or unfamiliar. The review therefore emphasizes the need for stronger benchmarks that measure long-context understanding, emotional interpretation, open-ended reasoning, sound localization and reliability under changing acoustic conditions. Evaluations based only on short clips or predefined labels may not reflect how systems behave in homes, vehicles, workplaces or public spaces.
Audio–visual integration offers another major route toward richer machine intelligence. Sound and vision provide complementary evidence: a camera may show a person opening a door while a microphone captures a knock, a warning alarm or a response from outside the frame. Combining the modalities can help a system determine where an event occurred, identify which visible object produced a sound and interpret scenes that would be ambiguous through one sensory channel alone. This requires cross-modal alignment, because the timing of an acoustic signal may not exactly match the appearance of its source. It also requires reasoning about absence. A loud sound with no visible source, or a visible action with no expected acoustic consequence, can both be important clues.
The researchers argue that progress will depend on models able to move fluidly among perception, reasoning and action. General auditory intelligence would need to recognize sounds, represent their temporal and social meaning, communicate uncertainty and use the information to make decisions. It would also need to avoid confident misinterpretations, an especially serious concern when audio is used in healthcare, accessibility tools, industrial monitoring or emergency response. Privacy presents another challenge because microphones can capture intimate conversations and sensitive background information. Robust systems will require careful data governance, transparent evaluation and safeguards against unauthorized recording, voice imitation and the misuse of generated speech.
The review presents the field as an important step toward embodied artificial intelligence: machines that do not merely process words or images, but participate in the sensory world through listening and speaking. If current research succeeds, future systems could understand complex acoustic scenes, generate more expressive sounds and hold conversations that preserve the subtle cues people use every day. Yet the authors stress that general auditory intelligence remains an open scientific problem. Machines still struggle with common-sense interpretation, unfamiliar sounds, long-duration events and the social meaning of voice. Solving those problems could make audio a central foundation of naturalistic machine interaction—and transform the way artificial systems perceive, reason about and respond to the world.
Subject of Research: General auditory intelligence for machine listening, audio comprehension, audio generation, speech-based interaction and audio–visual understanding.
Article Title: Towards general auditory intelligence for machine listening and speaking
Article References: Wang, S., Jin, Z., Tang, C. et al. “Towards general auditory intelligence for machine listening and speaking.” Nature Machine Intelligence (2026). https://doi.org/10.1038/s42256-026-01281-1
Image Credits: AI Generated
DOI: https://doi.org/10.1038/s42256-026-01281-1
Keywords: computer audition, auditory intelligence, large language models, audio comprehension, audio generation, speech interaction, multimodal AI, audio–visual understanding, machine listening, artificial intelligence
