The motivation comes from a gap the authors identify in a rapidly maturing industry. Streaming automatic speech recognition, low-latency neural text-to-speech, and zero-shot voice cloning have all reached production quality in the last few years, and commercial systems now dub live sports and broadcasts in real time. CAMB.AI’s DubStream dubs Major League Soccer and Ligue 1 matches, Microsoft Edge ships native real-time video translation, and YouTube rolled out auto-dubbing to all creators in February 2026. But each of these offerings is locked to its own surface. Edge’s feature works only inside Edge, YouTube’s dubbing is creator-gated and limited to video-on-demand uploads in 27 languages, and Windows Live Captions produce text but no dubbed audio. No existing system lets a viewer translate whatever they want, wherever they want, and none publishes its architecture, latency budget, or synchronization mechanism for the research community to study.

Polyglot closes those gaps with a two-part design: a Chromium Manifest V3 extension on the user’s machine and a stateless Python backend that orchestrates three cloud services. The extension is split into four cooperating browser contexts, each required by Chrome’s security model. A side panel hosts the interface with language pickers and color-coded per-speaker captions; a service worker coordinates the session; an offscreen document handles all audio and network work, downsampling captured tab audio to 16-kilohertz PCM in 200-millisecond chunks; and a content script injected into the page manages video synchronization. The backend is a FastAPI WebSocket server that streams the audio to Deepgram’s Nova-3 speech recognition, translates the resulting transcript with Google Translate, and synthesizes the dub with ElevenLabs’ Flash v2.5 text-to-speech model. Each service sits behind a thin adapter, so any one can be swapped without touching the rest of the pipeline. The default translation path is free and requires no API key, and the backend can run locally at no hosting cost or be deployed to the user’s own Modal account.

The most delicate engineering problem is speaker awareness. Deepgram’s diarization labels drift over long sessions and occasionally merge two speakers into one, so Polyglot recomputes a per-utterance speaker fingerprint using ECAPA-TDNN, a learned speaker-embedding model from the speaker-verification literature, and clusters utterances into stable identities downstream. Each detected speaker is then matched to one of roughly 25 stock ElevenLabs voices by estimated pitch, gender, and energy, with the assignment locked once about three seconds of that speaker’s audio has accumulated. When a new voice appears mid-broadcast, as when a sports feed cuts to a sideline interview, playback briefly pauses in a re-buffer phase while the analysis completes, with a five-second timeout preventing indefinite stalls. The authors validated the matching heuristic with Speaker Embedding Cosine Similarity, a standard voice-cloning metric: on a multi-speaker TEDx clip, matched voices scored a mean similarity of 0.108 against the source speakers, roughly four times the 0.026 obtained from random assignment, showing the heuristic picks up genuine acoustic signal.

Synchronization is the second hard problem, and the one the authors say the field has largely ignored. A cascaded pipeline takes several seconds to produce translated audio, so if the dub simply plays over an undelayed video, the picture races ahead by the full pipeline latency, typically five to twelve seconds, far outside the roughly two-second tolerance for audio-late stimuli established in classic human-perception studies. Polyglot’s primary fix is a canvas overlay: for unprotected video, the content script hides the original player, captures every rendered frame into a rolling buffer using the browser’s requestVideoFrameCallback API, and redraws each frame onto an overlay canvas exactly the measured pipeline delay late. The viewer sees a frame-accurate copy of the video, shifted in time to match the dub. For DRM-protected players such as Netflix, where the browser cannot read protected frames, the extension detects the black canvas and falls back to a one-shot seek-back that rewinds the player by the buffer duration. In both modes, a drift-correction loop nudges the video’s playback rate between 0.65 and 1.15 times normal speed, inaudibly, whenever picture and audio diverge.

The empirical evaluation is, to the authors’ knowledge, the first of an in-browser speech-to-speech translation system to measure screen-versus-audio offset directly. Translation quality was assessed with the SimulEval toolkit on four English-source pairs spanning three language families and two scripts: Spanish, French, and German from the Europarl-ST corpus of parliamentary speech, and Hindi from the FLEURS evaluation set. Text-level COMET-22, a neural metric that correlates best with human judgments, landed between 0.66 and 0.73 across the European pairs, in the mid-tier production range. Crucially, the drop from text-level to speech-level quality, which includes the text-to-speech synthesis and a Whisper re-transcription round-trip, was at most 0.011 on the European pairs, meaning the dub preserves nearly all the semantic content of the translation. The Hindi pair showed a wider gap, which the authors attribute to Whisper’s weaker accuracy on synthesized Devanagari speech rather than to the dub itself.

On synchronization, in-browser telemetry sampled the offset between the frame the viewer sees and the audio position they hear at five to ten hertz. Across three English-to-Spanish sessions on the TEDx clip, the drift-correction loop held mean offset between 1.2 and 3.5 seconds, with the best session averaging 1.23 seconds and a 99th percentile under two seconds, inside the classical acceptability bound. The canvas-overlay path itself contributes zero variance, so all residual offset comes from the controller compensating for pipeline jitter. Translation latency, measured as Average Lagging, came to roughly six seconds on the European pairs and 9.4 seconds on Hindi, dominated by the speech recognizer’s silence-end finalization and the text-to-speech model’s first-chunk synthesis time. The authors note that cutting latency further would require replacing the cascaded cloud architecture with an end-to-end streaming model, but they emphasize that latency shifts the experience a few seconds behind live without affecting whether the dub is watchable.

Beyond the headline capability, the system is designed for robustness and modest cost. The frame buffer is hard-capped at about 300 frames, roughly 469 megabytes, and the overlay loop costs about 0.55 milliseconds per frame, under two percent of one CPU core at 30 frames per second. Network traffic is a single WebSocket carrying about 256 kilobits per second upstream and 64 downstream. Running costs are pass-through usage on the user’s own backend: at current self-serve prices, speech recognition costs about $0.29 per hour of audio while text-to-speech dominates at roughly $4 per hour, with translation free by default. A second capture mode runs the same pipeline on microphone input, producing a downloadable, caption-synchronized translated voice memo for short cross-language conversations, with live playback suppressed to avoid a feedback loop. Coverage spans 19 languages in the intersection of the three services, yielding 342 directional language pairs.

The authors are candid about limitations. The synchronization evaluation focused on talking-head video, end-to-end latency is bounded by the three cloud services, the seek-back fallback for DRM content is perceptually inferior to the canvas overlay, and the translation tests ran at smoke scale on four target languages. The extension also runs only on desktop Chromium, since the required side-panel, offscreen-document, and tab-capture APIs are unavailable on mobile. Future work includes streaming wait-k translation models to push latency below one second, Bluetooth latency compensation, isochronous text-to-speech for non-talking-head content, and per-speaker voice cloning, which the voice-agnostic synthesis layer can already accommodate without upstream changes. The team also plans a Chrome Web Store listing with a hosted backend so non-technical users can install the extension without configuring API keys.

For now, Polyglot stands as an open, citable counterpart to a category of closed, business-to-business dubbing systems. By publishing the full latency budget, the synchronization telemetry, and reusable components including the streaming message protocol, the speaker re-clustering layer, and the canvas frame-delay technique, the authors give researchers in streaming translation, automatic dubbing, and web platforms a working codebase to read, replicate, and build upon. And for viewers, it shifts a quiet power: what gets translated, and into which language, is decided no longer by a platform or a content owner, but by the person watching.

Subject of Research: Real-time, speaker-aware speech-to-speech translation of web video in the browser

Article Title: Polyglot: An open-source browser extension for real-time, speaker-aware speech-to-speech translation of web video

Article References: Sandler, M., & Agarwal, P. (2026). Polyglot: An open-source browser extension for real-time, speaker-aware speech-to-speech translation of web video. SoftwareX, 36, Article 103026. https://doi.org/10.1016/j.softx.2026.103026

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103026

Keywords: speech-to-speech translation, browser extension, real-time dubbing, speaker diarization, audio-video synchronization, streaming speech recognition, text-to-speech synthesis, voice matching, machine translation, open-source software, Polyglot, open-source