The motivation comes from a gap the authors identify in a rapidly maturing industry. Streaming automatic speech recognition, low-latency neural text-to-speech, and zero-shot voice cloning have all reached production quality in the last few years, and commercial systems now dub live sports and broadcasts in real time. CAMB.AI’s DubStream dubs Major League Soccer and Ligue 1 matches, Microsoft Edge ships native real-time video translation, and YouTube rolled out auto-dubbing to all creators in February 2026. But each of these offerings is locked to its own surface. Edge’s feature works only inside Edge, YouTube’s dubbing is creator-gated and limited to video-on-demand uploads in 27 languages, and Windows Live Captions produce text but no dubbed audio. No existing system lets a viewer translate whatever they want, wherever they want, and none publishes its architecture, latency budget, or synchronization mechanism for the research community to study.

Polyglot closes those gaps with a two-part design: a Chromium Manifest V3 extension on the user’s machine and a stateless Python backend that orchestrates three cloud services. The extension is split into four cooperating browser contexts, each required by Chrome’s security model. A side panel hosts the interface with language pickers and color-coded per-speaker captions; a service worker coordinates the session; an offscreen document handles all audio and network work, downsampling captured tab audio to 16-kilohertz PCM in 200-millisecond chunks; and a content script injected into the page manages video synchronization. The backend is a FastAPI WebSocket server that streams the audio to Deepgram’s Nova-3 speech recognition, translates the resulting transcript with Google Translate, and synthesizes the dub with ElevenLabs’ Flash v2.5 text-to-speech model. Each service sits behind a thin adapter, so any one can be swapped without touching the rest of the pipeline. The default translation path is free and requires no API key, and the backend can run locally at no hosting cost or be deployed to the user’s own Modal account.

The most delicate engineering problem is speaker awareness. Deepgram’s diarization labels drift over long sessions and occasionally merge two speakers into one, so Polyglot recomputes a per-utterance speaker fingerprint using ECAPA-TDNN, a learned speaker-embedding model from the speaker-verification literature, and clusters utterances into stable identities downstream. Each detected speaker is then matched to one of roughly 25 stock ElevenLabs voices by estimated pitch, gender, and energy, with the assignment locked once about three seconds of that speaker’s audio has accumulated. When a new voice appears mid-broadcast, as when a sports feed cuts to a sideline interview, playback briefly pauses in a re-buffer phase while the analysis completes, with a five-second timeout preventing indefinite stalls. The authors validated the matching heuristic with Speaker Embedding Cosine Similarity, a standard voice-cloning metric: on a multi-speaker TEDx clip, matched voices scored a mean similarity of 0.108 against the source speakers, roughly four times the 0.026 obtained from random assignment, showing the heuristic picks up genuine acoustic signal.

Synchronization is the second hard problem, and the one the authors say the field has largely ignored. A cascaded pipeline takes several seconds to produce translated audio, so if the dub simply plays over an undelayed video, the picture races ahead by the full pipeline latency, typically five to twelve seconds, far outside the roughly two-second tolerance for audio-late stimuli established in classic human-perception studies. Polyglot’s primary fix is a canvas overlay: for unprotected video, the content script hides the original player, captures every rendered frame into a rolling buffer using the browser’s requestVideoFrameCallback API, and redraws each frame onto an overlay canvas exactly the measured pipeline delay late. The viewer sees a frame-accurate copy of the video, shifted in time to match the dub. For DRM-protected players such as Netflix, where the browser cannot read protected frames, the extension detects the black canvas and falls back to a one-shot seek-back that rewinds the player by the buffer duration. In both modes, a drift-correction loop nudges the video’s playback rate between 0.65 and 1.15 times normal speed, inaudibly, whenever picture and audio diverge.

The empirical evaluation is, to the authors’ knowledge, the first of an in-browser speech-to-speech translation system to measure screen-versus-audio offset directly. Translation quality was assessed with the SimulEval toolkit on four English-source pairs spanning three language families and two scripts: Spanish, French, and German from the Europarl-ST corpus of parliamentary speech, and Hindi from the FLEURS evaluation set. Text-level COMET-22, a neural metric that correlates best with human judgments, landed between 0.66 and 0.73 across the European pairs, in the mid-tier production range. Crucially, the drop from text-level to speech-level quality, which includes the text-to-speech synthesis and a Whisper re-transcription round-trip, was at most 0.011 on the European pairs, meaning the dub preserves nearly all the semantic content of the translation. The Hindi pair showed a wider gap, which the authors attribute to Whisper’s weaker accuracy on synthesized Devanagari speech rather than to the dub itself.

On synchronization, in-browser telemetry sampled the offset between the frame the viewer sees and the audio position they hear at five to ten hertz. Across three English-to-Spanish sessions on the TEDx clip, the drift-correction loop held mean offset between 1.2 and 3.5 seconds, with the best session averaging 1.23 seconds and a 99th percentile under two seconds, inside the classical acceptability bound. The canvas-overlay path itself contributes zero variance, so all residual offset comes from the controller compensating for pipeline jitter. Translation latency, measured as Average Lagging, came to roughly six seconds on the European pairs and 9.4 seconds on Hindi, dominated by the speech recognizer’s silence-end finalization and the text-to-speech model’s first-chunk synthesis time. The authors note that cutting latency further would require replacing the cascaded cloud architecture with an end-to-end streaming model, but they emphasize that latency shifts the experience a few seconds behind live without affecting whether the dub is watchable.

Beyond the headline capability, the system is designed for robustness and modest cost. The frame buffer is hard-capped at about 300 frames, roughly 469 megabytes, and the overlay loop costs about 0.55 milliseconds per frame, under two percent of one CPU core at 30 frames per second. Network traffic is a single WebSocket carrying about 256 kilobits per second upstream and 64 downstream. Running costs are pass-through usage on the user’s own backend: at current self-serve prices, speech recognition costs about $0.29 per hour of audio while text-to-speech dominates at roughly $4 per hour, with translation free by default. A second capture mode runs the same pipeline on microphone input, producing a downloadable, caption-synchronized translated voice memo for short cross-language conversations, with live playback suppressed to avoid a feedback loop. Coverage spans 19 languages in the intersection of the three services, yielding 342 directional language pairs.

The authors are candid about limitations. The synchronization evaluation focused on talking-head video, end-to-end latency is bounded by the three cloud services, the seek-back fallback for DRM content is perceptually inferior to the canvas overlay, and the translation tests ran at smoke scale on four target languages. The extension also runs only on desktop Chromium, since the required side-panel, offscreen-document, and tab-capture APIs are unavailable on mobile. Future work includes streaming wait-k translation models to push latency below one second, Bluetooth latency compensation, isochronous text-to-speech for non-talking-head content, and per-speaker voice cloning, which the voice-agnostic synthesis layer can already accommodate without upstream changes. The team also plans a Chrome Web Store listing with a hosted backend so non-technical users can install the extension without configuring API keys.

For now, Polyglot stands as an open, citable counterpart to a category of closed, business-to-business dubbing systems. By publishing the full latency budget, the synchronization telemetry, and reusable components including the streaming message protocol, the speaker re-clustering layer, and the canvas frame-delay technique, the authors give researchers in streaming translation, automatic dubbing, and web platforms a working codebase to read, replicate, and build upon. And for viewers, it shifts a quiet power: what gets translated, and into which language, is decided no longer by a platform or a content owner, but by the person watching.

Subject of Research: Real-time, speaker-aware speech-to-speech translation of web video in the browser

Article Title: Polyglot: An open-source browser extension for real-time, speaker-aware speech-to-speech translation of web video

Article References: Sandler, M., & Agarwal, P. (2026). Polyglot: An open-source browser extension for real-time, speaker-aware speech-to-speech translation of web video. SoftwareX, 36, Article 103026. https://doi.org/10.1016/j.softx.2026.103026

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103026

Keywords: speech-to-speech translation, browser extension, real-time dubbing, speaker diarization, audio-video synchronization, streaming speech recognition, text-to-speech synthesis, voice matching, machine translation, open-source software, Polyglot, open-source

The motivation behind the toolkit is rooted in a long-standing tension within the field. Laboratory speech, produced in word-reading or sentence-repetition tasks, gives experimenters precise control over variables and supports repeatability, which is why it has anchored theoretical work in speech perception and production for generations. But an accumulating body of evidence shows that listeners and speakers dynamically shape one another in everyday conversation, adapting to accents, talkers, and social-linguistic associations in ways that rigid laboratory tasks cannot capture. The classic observer’s paradox, articulated by William Labov in 1972, compounds the problem: people who know they are being recorded often adjust their speech, consciously or not, making genuinely naturalistic samples scarce and difficult to obtain.

The logistical hurdles do not end there. Field recordings demand extensive preparation, community relationships, and careful attention to the researcher’s positionality. Reading tasks systematically exclude speakers — such as many heritage bilinguals — who are conversationally fluent but may lack literacy in the language being studied. Funding constraints make it increasingly difficult to build and share open-access speech corpora, a burden that falls hardest on graduate students, early-career scholars, and researchers working on minoritized language communities. The authors argue that these barriers do not merely slow research down; they distort the theoretical record, because dominant models of speech perception and production were built on idealized, read speech and often treat real-world linguistic diversity as noise rather than as the phenomenon itself.

TAPA’s answer is to stitch together a chain of proven open-source components into one Python pipeline that can be installed with a single command and run either locally or in Google Colab, where free GPU access removes the need for any local setup. The journey begins with a YouTube audio downloader built on yt-dlp, which pulls the audio stream from any standard video URL and converts it to an MP3 file; users can equally supply their own local audio. Transcription is handled by OpenAI’s Whisper, which produces word-level transcripts with precise start and end timestamps. Speaker diarization — deciding who spoke when — relies on two pretrained neural models: Silero VAD, which detects stretches of actual speech, and Resemblyzer, which maps each detected segment to a 256-dimensional voice embedding in which clips from the same speaker cluster together. Those embeddings are then grouped to assign every transcribed word to its speaker.

From there, the pipeline turns to fine-grained phonetic measurement. The Montreal Forced Aligner produces phone-level boundaries linking the audio to the transcript, using an American English acoustic model with a dictionary-based proportional-timing fallback when alignment is unavailable. Vowel formants — the resonant frequencies F1 and F2 that define vowel identity — are extracted with Praat-parselmouth, using the Burg algorithm over the central portion of each vowel after trimming the edges to suppress coarticulation, with the maximum formant ceiling adjusted adaptively to each speaker’s pitch. Stop consonant voice onset time, the interval between a stop’s burst release and the onset of voicing, is measured by Dr. VOT, a neural model that predicts burst and voicing boundaries directly from the waveform. Fricative spectral moments — center of gravity, standard deviation, skewness, and kurtosis — are computed over the middle 80 percent of each fricative interval. The output is a set of CSV or JSON files containing per-token measurements and per-speaker summaries.

To demonstrate what the pipeline can do, the team pointed it at a demanding piece of real-world audio: the 90-minute 2016 U.S. presidential debate between Hillary Clinton and Donald Trump, complete with audience noise, podium reverberation, overlapping talkers, and the performative speech style of political theater. From that single recording, TAPA extracted nearly 33,000 acoustic segments — 19,855 vowels, 6,547 stops, and 6,372 fricatives. The case study focused on the two main candidates, yielding more than 7,000 monophthong tokens each for vowel-space analysis, alongside thousands of stop and fricative tokens per speaker. Plotting the F1-by-F2 vowel spaces revealed that Clinton’s vowel categories occupied a larger acoustic area than Trump’s, and comparing both speakers’ category means against the classic Hillenbrand reference norms showed the expected centralization of vowels in continuous speech relative to citation-form productions.

Crucially, the team did not simply trust the automated output. A phonetically trained research assistant hand-coded stratified samples of vowels, stops, and fricatives from the debate audio, blind to the pipeline’s results, and the two sets of measurements were compared token by token. Vowel formants emerged as the pipeline’s strongest suit: agreement with expert measurements was high, with correlations of 0.89 for F1 and 0.87 for F2, mean absolute errors of 37 and 125 hertz respectively, and no meaningful bias for F1. Fricative measurements showed a more differentiated picture. Spectral standard deviation agreed strongly with hand coding overall, and center of gravity tracked the human reference closely for the sibilant fricatives — correlations of 0.87 for /s/ and 0.94 for /ʃ/ — while the spectrally indistinct non-sibilants /f/ and /θ/ agreed far more weakly, a pattern consistent with long-standing findings that these sounds are hard to classify from spectral moments alone. Clinton’s /s/ center of gravity sat roughly 900 hertz above Trump’s, echoing documented patterns of gender-linked variation in sibilant production.

Stop consonants told a more cautionary tale. Although the aggregate voice-onset-time means for voiceless stops /p/, /t/, and /k/ landed squarely in the expected American English long-lag range of roughly 69 to 83 milliseconds, the per-token picture was far less reassuring. The Dr. VOT model, which was trained on isolated single-word productions with stop classes artificially rebalanced to a 50:50 positive-to-negative ratio, mislabeled 45 to 61 percent of voiceless onset stops as prevoiced — an outcome incompatible with English phonology — and its per-token magnitudes correlated essentially not at all with expert measurements (r = −0.04), diverging by 50 to 80 milliseconds on average. The authors attribute this to a training–deployment mismatch: continuous, noisy debate audio with a natural 95:5 ratio of positive to negative stops violates both of the model’s core training assumptions. Rather than hide the problem, they report it transparently, omit voiced stops from the case study entirely, and recommend that aggregate means be treated as descriptive rather than per-token measurements.

That candor reflects the paper’s broader stance on automation in science. The authors are explicit that TAPA is meant to aid researchers, not replace them: acoustic training must continue, research must be deployed and interpreted by humans, and the energy costs of training and fine-tuning large models warrant critical scrutiny. The pipeline inherits the biases of its components — the default configuration is American English-centric, diarization degrades on heavily overlapping speech, and forced alignment stumbles on out-of-vocabulary names — so the team recommends that every substantive deployment include a small hand-coded validation subsample from the researcher’s own corpus, and that publications report which component produced each measurement. With those safeguards in place, TAPA offers something the field has lacked: a free, modular, inspectable path from a web video or field recording to speaker-level, phone-level acoustic data at a scale that would once have required years of manual annotation, potentially opening naturalistic speech research to communities and questions that laboratory methods have long left on the periphery.

Subject of Research: An open-source computational pipeline for automated acoustic-phonetic analysis of naturalistic multi-speaker speech recordings

Article Title: Toolkit for acoustic–phonetic analysis of naturalistic speech data

Article References: Kutlu, E., Peters, E., Tapanes, C., Chandio, S., & Khalid, O. (2026). Toolkit for acoustic–phonetic analysis of naturalistic speech data. Behavior Research Methods, 58(10), Article 289. https://doi.org/10.3758/s13428-026-03174-y

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03174-y

Keywords: acoustic-phonetic analysis, naturalistic speech, speaker diarization, forced alignment, vowel formants, voice onset time, fricative spectral moments, open-source software, speech perception, sociophonetics, automatic speech recognition, Behavior Research Methods