Papers › Deep Audio-Visual Speech Recognition

Deep Audio-Visual Speech Recognition

6 Sep 2018arXiv:1809.02108archive 2025-07-28

Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

amitai1992/AutomatedLipReading mentioned on GitHubSyntology: not harvested report
exgc/avmust-ted mentioned on GitHubMITSyntology: not harvested report
lordmartian/deep_avsr mentioned on GitHubpytorchMITSyntology: not harvested report
smeetrs/deep_avsr mentioned on GitHubpytorchMITSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Audio-Visual Speech RecognitionAutomatic Speech Recognition (ASR)Lip ReadingLipreadingSpeech RecognitionVisual Speech Recognitionspeech-recognition

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Audio-Visual Speech Recognition LRS2 TM-CTC Test WER 8.2 #7 of 8 Archive leaderboard report
Audio-Visual Speech Recognition LRS2 TM-Seq2seq Test WER 8.5 #8 of 8 Archive leaderboard report
Audio-Visual Speech Recognition LRS3-TED TM-seq2seq Word Error Rate (WER) 7.2 #12 of 12 Archive leaderboard report
Automatic Speech Recognition (ASR) LRS2 TM-seq2seq Test WER 9.7 #8 of 9 Archive leaderboard report
Automatic Speech Recognition (ASR) LRS2 TM-CTC Test WER 10.1 #9 of 9 Archive leaderboard report
Lipreading LRS2 TM-seq2seq + extLM Word Error Rate (WER) 48.3 #19 of 25 Archive leaderboard report
Lipreading LRS2 TM-CTC + extLM Word Error Rate (WER) 54.7 #24 of 25 Archive leaderboard report
Lipreading LRS3-TED TM-seq2seq Word Error Rate (WER) 58.9 #21 of 23 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

CTC Loss

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections