Lip reading in 2026: how good the models are, and how far silent speech still has to go (Oct 2026)

Machine lip reading has improved quickly. In 2016 LipNet could only read short, fixed-grammar sentences. Today the best open model, USR 2.0, gets about 18% of words wrong on the standard TED-talk benchmark, and Google's private models report under 13%. Those numbers sound close to useful dictation. They are not, for three reasons. The benchmark is easier than real video. Silent speech is harder than the voiced speech the models learn from. And every model gets worse on a face it has never seen.

At the same time, the commercial interest has moved from cameras to sensors. Apple paid between $1.6 billion and $2 billion for Q.ai, a startup that reads "facial skin micromovements". AlterEgo is building a jaw-mounted EMG headset. Meta has shown a wrist EMG band that decodes handwriting.

	Value	Change
Best LRS3 lip reading, private data	12.5–12.8% WER
Best open model (USR 2.0 Huge)	17.6% WER
Same kind of model on harder video (WildVSR)	~40% WER
Silent EMG, open vocabulary (MONA LISA)	12.2% WER
Silent webcam dictation, off the shelf	45–62% WER

The benchmark everyone reports, and why it flatters

Almost every lip-reading paper reports word error rate (WER) on LRS3: about 400 hours of TED and TEDx clips, with a test set of roughly one hour. The speakers are well lit, mostly facing the camera, and talking out loud to an audience. They are also professional, practised speakers.

That makes LRS3 a good benchmark for comparing methods, and a poor estimate of real-world accuracy. In 2023, researchers at the Technology Innovation Institute rebuilt an LRS3-style test set from new videos and called it WildVSR. Publicly available models lost an average of about 30 WER points on it, though their ranking stayed mostly the same (Djilali et al., WACV 2024). On single words the gap is starker: models that score around 90% on the LRW word benchmark score 30–35% on WildVSR's word version.

The gap has not closed. A 2026 paper using the USR 2.0 encoder reports 19.4% WER on LRS3 and 39.9% on WildVSR, about double (diffusion LLMs for VSR, 2026). A second 2026 paper, "The Lipreading Gap", argues that models trained on LRS3 lean on dataset-specific patterns rather than reading lips the way people do (arXiv 2606.07435).

Note

A WER of 20% means one word in five is wrong. For scale, the same USR 2.0 Huge model gets 0.9% WER when it hears the audio of the LRS3 clips instead of watching the lips (USR 2.0 release).

The leaderboard

Results depend heavily on how much labelled video a model saw, so the table separates private-data and public-data systems.

System	Year	LRS3 WER	Training data	Weights
Google linear-front-end Conformer (two reported versions)	2023	12.5–12.8%	Large private YouTube corpus	No
USR 2.0 Huge	2026	17.6%	LRS2, LRS3, VoxCeleb2, AVSpeech (semi-supervised)	Yes, MIT
VALLR (phonemes + LLM)	2025	18.7%	LRS3 labels only	Promised, not found
Auto-AVSR (largest)	2023	19.1%	Over 3,000 h with ASR-generated labels	Yes, benchmarking use
RAVEn Large + self-training	2023	19.3%	LRS3 + VoxCeleb2	Yes
Diffusion LLM + USR 2.0 encoder	2026	19.4%	LRS3 labels only	—
Auto-AVSR (base, 3,291 h)	2023	20.3%	~3,300 h with ASR-generated labels	Yes, benchmarking use
USR 2.0 Large	2026	21.5%	LRS3 + VoxCeleb2	Yes, MIT
Llama-2-13B decoder (combined data)	2025	24.7%	Combined public sets	—

Lip reading on LRS3 (lower is better)

	WER
Google Conformer	12.5%
USR 2.0 Huge	17.6%
VALLR	18.7%
Auto-AVSR 19.1	19.1%
Diffusion LLM	19.4%
Auto-AVSR base	20.3%
USR 2.0 Large	21.5%
LLM decoder 13B	24.7%

The Google results come from Chang et al. 2023, which reports 12.8% and argues that a simple linear front end and a bigger Conformer encoder beat elaborate visual front ends; a 12.5% figure for the same linear-projection design is cited in the VALLR comparison tables (VALLR, ICCV 2025). USR 2.0's numbers come from its code release and paper. Auto-AVSR's come from its repository and paper.

The best open model, USR 2.0 Huge, is about five points behind Google's private systems. The gap is mostly data: Google trained on far more labelled YouTube video than any public set offers.

Four ideas behind the progress

1. Let speech recognisers label the video

Hand-transcribed lip-reading video is scarce. Auto-AVSR's insight was to run a strong audio recogniser over thousands of hours of unlabelled talking-head video and train the lip reader on those automatic transcripts (Ma et al., ICASSP 2023). Going from 438 to 3,291 hours took its WER from 36% to 20.3%. Almost every strong open model since then uses some form of this pseudo-labelling.

2. One model for audio, video, or both

AV-HuBERT, RAVEn and BRAVEn pre-train on unlabelled video by predicting audio-derived targets. USR (NeurIPS 2024) went further and trained one model that handles audio only, video only, or both, sharing what it learns between them (Haliassos et al.). USR 2.0 (ICLR 2026) keeps that design but generates its pseudo-labels with a single CTC pass instead of slow step-by-step decoding. That halves training time and, the authors report, makes the model more robust to video unlike its training data (arXiv 2602.19316). The paper also shows that CTC decoding holds up better than attention decoding on out-of-domain and long inputs, while attention decoding wins on familiar data.

3. Large language models as decoders

The newest wave attaches an LLM to a visual encoder. VSP-LLM feeds AV-HuBERT features into LLaMA-2 (arXiv 2402.15151). Llama-AVSR keeps the encoders and the LLM frozen and trains only small adapters; it is roughly on par with the best non-LLM systems for lip reading and excellent for audio-visual recognition (arXiv 2409.12319). MMS-LLaMA cuts the token cost by about 86% (arXiv 2503.11315). VALLR predicts phonemes from video and lets a fine-tuned LLM rebuild the words, reaching 18.7% with far less labelled data than its rivals (ICCV 2025). A 2026 preprint swaps the usual left-to-right LLM for a diffusion LLM and reaches 19.4% using only LRS3 labels (arXiv 2605.28456).

A careful 2025 study checked what the LLM actually contributes. Its answer: mostly context and language knowledge, not better reading of the lips (arXiv 2509.14880). That fits the physics. Many sounds look the same on the lips (p, b and m; f and v; t, d and n), and k and g happen out of sight. Only context can separate "pat", "bat" and "mat".

4. Borrow a speech recogniser's brain

Oxford's group showed that a frozen Whisper model can read lips if a small network learns to map lip movements into Whisper's speech representations. It gave a 5.7-point WER gain over earlier methods when labelled data is scarce (Prajwal, Afouras and Zisserman, Interspeech 2024). The idea is the same as with LLM decoders: reuse a model that already knows language, and teach it only the visual part.

Silent speech is a harder problem

Every model above learned from people speaking out loud. Silent speech, mouthing words without sound, looks different. Without air pressure in the mouth, people move their lips and jaw less, and sounds like /m/ barely show at all (Lip-Siri, 2026).

The measured gap varies by study. On a small-vocabulary task, models trained on normal speech lost up to 8.5 points of accuracy on silent speech, against 3.7 points on whispered speech (Petridis et al., ICASSP 2018). Kashiwagi et al. narrowed the gap with viseme-based metric learning, even with little silent training data (Interspeech 2023). In a 2025 EMG neckband study, accuracy fell from 87% for vocalised to 68% for silent articulation (arXiv 2509.21964). There is still no large public dataset of silent, open-vocabulary video to train on.

The other big gap is the speaker. Models get noticeably worse on faces they have not seen, and adapting them to one person helps. Yeo et al. adapt both to a speaker's lip movements and to their vocabulary, and report larger gains than earlier visual-only adaptation (AAAI 2025).

A field test: silent dictation through a webcam

To see how these numbers translate, a small proof of concept was built on an Apple M5 Max Mac (CPU inference): a browser records webcam clips, Auto-AVSR or USR 2.0 reads the lips, and an LLM rewrites the output, given the lip reader's top five guesses. One speaker mouthed eight everyday sentences silently ("The package should arrive sometime before noon tomorrow", "Please remind me to call the doctor on Thursday") in a dim room, sitting at normal laptop distance. Each was scored against what was said.

Pipeline	WER
Auto-AVSR (20.3% on LRS3), top guess	62.2%
USR 2.0 Huge (17.6% on LRS3), top guess	64.6%
Auto-AVSR + Claude Haiku 5.5 cleanup	53.7%
Auto-AVSR + GPT-6 Luna cleanup	52.4%
Auto-AVSR + Qwen 3.8 Max cleanup	45.1%
Auto-AVSR, best of its top 5 guesses	45.1%

Silent webcam dictation, 8 sentences, one speaker (lower is better)

	WER
USR 2.0 Huge	64.6%
Auto-AVSR	62.2%
+ Haiku 5.5	53.7%
+ GPT-6 Luna	52.4%
+ Qwen 3.8 Max	45.1%

The sample is small (82 words), so a few points either way are noise. Several things still stand out:

Benchmark rank did not carry over. USR 2.0 Huge beats Auto-AVSR by 2.7 points on LRS3, but was no better here. It was also 2–3 times slower on a CPU, needed 12 GB of memory, and its beam search took 55 seconds per 5-second clip until the beam was cut from 10 to 2.
Errors tripled compared with the benchmark. Both models landed above 60% WER, three times their LRS3 numbers. The likely causes are the ones above: silent articulation, an unfamiliar face, and a dim, small face in the frame. The mouth crops averaged 20–50 out of 255 in brightness, against about 86 for a well-lit broadcast clip. Rescaling the crops to the training brightness made the output noticeably more coherent but did not fix it.
The LLM matters more than the lip reader. The best cleanup model cut WER by 17 points, and the gap between cleanup models was 9 points; switching lip readers changed WER by 2. The best LLM rebuilt "the bandage should arrive september 4th through tomorrow" into exactly "The package should arrive sometime before noon tomorrow". It roughly matched picking the best of the lip reader's five guesses every time.
Context leaks are easy. In one clip the intended sentence had been typed into a context field, and the LLM produced it from garbled input. Any evaluation of LLM-corrected lip reading has to keep hints out.

Beyond the camera: muscles, sound, radio and brains

Cameras have a basic limit: they cannot see the tongue or the throat. The strongest silent-speech results come from sensors that can.

Surface EMG on the face and neck. Gaddy and Klein's 2020 work trained on voiced EMG and transferred to silent EMG by aligning the two recordings (Digital Voicing of Silent Speech). Their 2021 model cut open-vocabulary WER from 68% to 42% (arXiv 2106.01933). In 2024, MONA LISA brought silent EMG on the same benchmark to 12.2% WER, using cross-modal training and LLM-based correction, with 3.7% for voiced EMG (arXiv 2403.05583). That is better than any camera system on silent speech, but it relies on electrodes stuck to the face and on one speaker's data. A 2025 study shows a dry-electrode EMG neckband working at ultra-low power (arXiv 2509.21964), and emg2speech synthesises audio from EMG using self-supervised speech models (arXiv 2510.23969).

Meta's wrist band. Not speech, but the clearest sign that EMG can be productised. Meta's Nature paper shows a generic wrist sEMG model that lets new users handwrite at a median 20.9 words per minute, with personalisation adding 16% (Kaifosh et al., Nature 2025; The Register). The pattern of a model trained on many people, then tuned to one, is what silent-speech wearables will need too.

Acoustic and radio sensing. Cornell's EchoSpeech glasses bounce sound off the face to recognise silent commands (Science News Explores). Lip-Siri reads open sentences from Wi-Fi backscatter with no camera at all (arXiv 2601.18177).

Brain implants. At the far end, a Stanford-led BrainGate study decoded imagined speech from motor-cortex implants in four people with paralysis, with accuracy of up to 74% from a 125,000-word vocabulary. A mental password gated the decoder and was recognised 98% of the time (Cell Press release via Scimex; TechSpot). This is the only work that decodes truly internal speech, with no movement at all, and it requires surgery.

Who is building it

Apple and Q.ai. On January 29, 2026, Apple confirmed it had bought Q.ai, an Israeli startup founded in 2022. Reuters put the price at about $1.6 billion, while other reports said nearer $2 billion, which would make it Apple's second-largest acquisition after Beats. Apple described the work as machine learning to help devices understand whispered speech and to enhance audio in hard environments. A Q.ai patent application describes using "facial skin micromovements" to detect mouthed or spoken words and to read identity, emotion, heart rate and breathing. About 100 staff joined Apple, led by CEO Aviad Maizels, who previously founded PrimeSense (Yahoo Finance; Calcalist; TNW). Apple has not said how it will use the technology. Privacy critics point out that the same sensing can infer what people say or feel without any sound (The Cradle).

AlterEgo. The 2018 MIT Media Lab prototype became a company that unveiled a "near-telepathic" wearable on September 8, 2025. It hooks around the ear and runs along the jaw, reads faint neuromuscular signals when the wearer internally verbalises words, and answers through bone conduction. It does not read the brain. The company has not announced funding, pricing or a ship date (Decrypt; Fast Company SA).

Camera lip reading as a service. Symphonic Labs runs readtheirlips.com, which transcribes uploaded clips and is pitched at noisy calls and silent assistant commands; a reviewer found it mostly right on a 26-second clip (CO/AI). Smaller tools such as LipRead AI transcribe silent video (Product Hunt). None claims open-vocabulary silent dictation from a live webcam.

Open problems

The research points to the same handful of gaps, roughly in order of how much they would help silent dictation:

Silent, in-the-wild training data. Benchmarks and training sets are voiced, well-lit and mostly frontal. No public silent-speech video corpus exists at LRS3 scale.
Personalisation. Every system improves when tuned to one user, from Meta's wrist band to speaker-adaptive lip reading. A short enrolment session, reading prompted sentences, is the cheapest big win available.
Robustness beyond LRS3. WildVSR shows that LRS3 progress overstates real gains. USR 2.0's CTC-based training is one of the few methods aimed directly at out-of-domain video.
Context. Lip reading cannot separate look-alike words on its own. LLM decoders that also see what the user is writing, or what they said before, are where the next large drop in errors is likely to come from.
Sensors that see inside the mouth. EMG already reads silent speech better than cameras. The open question is whether it can be made comfortable and generic enough, which is what AlterEgo, and possibly Apple, are betting on.